Geoparsing Eval

geoparsing-eval

Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Language — Wang et al. (2026) (arXiv:2604.11600, 2026)

What this evaluates

This benchmark evaluates a model's ability to perceive and parse geometric diagrams into a structured formal language. It probes fine-grained visual primitive detection (points, lines, circles, planes) and spatial/semantic relations, testing both syntactic correctness and holistic geometric consistency.

Datasets

• GDP-29K — total 29000; splits: train (26000), test (3000)

Metrics

• F1-score (primary) — range: percent
  • Computed per primitive category by comparing predicted and ground-truth sets of geometric primitives and relations. F1 = 2 * (Precision * Recall) / (Precision + Recall).
• Sample Accuracy (SA) — range: percent
  • Binary metric indicating whether a single diagram is parsed perfectly as a whole. SA = 1 if the predicted formal description exactly matches the ground-truth description, else 0.
• Perfect Parsing Rate (PPR) — range: percent
  • The fraction of samples in a benchmark subset that achieve a Sample Accuracy of 1, measuring holistic diagram-level correctness.

Input / output format

Input: RGB images of plane or solid geometry diagrams.

Output: A unified formal language string representing the diagram's geometric primitives and spatial/semantic relations.

Scoring recipe

def compute_metrics(predictions, golds):
    p_list, r_list, f1_list, sa_list = [], [], [], []
    for pred_set, gold_set in zip(predictions, golds):
        tp = len(pred_set & gold_set)
        p = tp / len(pred_set) if pred_set else 0
        r = tp / len(gold_set) if gold_set else 0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        p_list.append(p); r_list.append(r); f1_list.append(f1)
        sa_list.append(1.0 if pred_set == gold_set else 0.0)
    return {
        'P': sum(p_list) / len(p_list),
        'R': sum(r_list) / len(r_list),
        'F1': sum(f1_list) / len(f1_list),
        'SA': sum(sa_list) / len(sa_list),
        'PPR': sum(sa_list) / len(sa_list)
    }

Common pitfalls

• High category-level F1 scores can mask holistic failures; a single primitive error invalidates the entire diagram's formal description (multiplier effect).
• Baseline models often detect basic primitives well but fail on higher-order semantic relations (e.g., parallelism, angles), requiring specialized training beyond general visual pre-training.
• Downstream reasoning gains may be underestimated on solid geometry benchmarks due to textual explicitness in problem statements, which reduces the marginal utility of parsed diagrams.

Evidence (verbatim from paper)

While category-level F1 measures fine-grained parsing quality, it does not necessarily indicate that a diagram is parsed perfectly as a whole. To better evaluate holistic correctness, we additionally report Sample Accuracy (SA) for each category and Perfect Parsing Rate (PPR) for the full diagram.

Citation

@misc{wang2026geoparsing,
  title={Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Language},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.11600}
}

• arXiv: 2604.11600