Polygonal masonry refers to a stone wall that was constructed through rubble masonry but in which the stones have been dressed to provide smooth(ish) faces and corners. This allows for closer fitting joints, more structural stability, and a better finished appearance.
It is perhaps worth noting that in this case, we are most certainly not refering to regular polygons; we are simply refering to highly irregular shapes with well defined corners and planes, which is not what we expect to see in most rock. Well done polygonal walls may look every bit as clean and modern as ashlar walls, despite being constructed of irregular shapes.