Forgotten Polygons Eval

forgotten-polygons-eval

Forgotten Polygons: Multimodal Large Language Models are Shape-Blind — Rudman et al. (2025) (arXiv:2502.15969, 2025)

What this evaluates

This benchmark probes the ability of multimodal large language models to recognize regular and irregular geometric shapes from images and accurately count their sides. It further evaluates multi-step visual-mathematical reasoning by requiring models to identify multiple shapes, map them to side counts, and compute their sum.

Datasets

• Forgotten Polygons — total 2000; splits: test (-1); repo https://github.com/rsinghlab/Shape-Blind

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted shape names or side counts out of the total number of instances. Calculated as (correct predictions / total instances) * 100.

Input / output format

Input: A 400×400 pixel image containing one or two geometric shapes (regular polygons, irregular polygons, or abstract shapes) with variations in color, rotation, size, and background. Accompanied by text prompts such as 'What shape is in the image?' or 'How many sides does the shape in the image have?'.

Output: Text response containing the shape name (e.g., 'triangle', 'pentagon') or an integer representing the number of sides.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if isinstance(gold, int):
            try:
                if int(pred) == gold: correct += 1
            except ValueError: pass
        else:
            if pred.strip().lower() == gold.strip().lower(): correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Confusing shape identification accuracy with side-counting accuracy, as models often recognize familiar shapes correctly but fail to count their sides (e.g., GPT-4o identifies heptagons at 92% but counts sides at 58%).
• Attributing low multi-step reasoning scores to arithmetic failures; the protocol shows Step 3 errors are primarily caused by incorrect shape identification or side mapping in earlier steps, not summation mistakes.
• Assuming language model backbones lack geometric knowledge; text-only prompts reveal near-perfect accuracy, indicating failures stem from vision encoders rather than textual pretraining gaps.

Evidence (verbatim from paper)

Table[1] shows that MLLMs excel at identifying common shapes like triangles and squares, with many models achieving perfect accuracy. However, performance drops sharply for less familiar polygons such as pentagons, heptagons, and octagons.

Citation

@misc{rudman2025forgottenpolygons,
  title={Forgotten Polygons: Multimodal Large Language Models are Shape-Blind},
  author={Rudman et al. (2025)},
  year={2025},
  note={arXiv:2502.15969}
}

• arXiv: 2502.15969