Ggbench Eval

ggbench-eval

GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models — Wei et al. (2025) (arXiv:2511.11134, 2025)

What this evaluates

Evaluates geometric generative reasoning in unified multimodal models by testing their ability to perform stepwise spatial planning, translate visual constraints into executable code or visual sequences, and verify multi-step geometric construction processes.

Datasets

• GGBench — total ?; splits: test (-1); repo https://github.com/opendatalab-raiser/GGBench

Metrics

• VLM-T (primary) — range: [1, 5]
  • Rubric-based scoring from 1 to 5 comparing generated textual construction steps against expert reference steps. Scores reflect logical completeness, geometric correctness, and stepwise coherence.
• VLM-I-Res — range: [1, 5]
  • Rubric-based scoring from 1 to 5 comparing the model's final rendered diagram against the reference solution. Scores reflect geometric consistency and constraint satisfaction.
• VLM-I-Mid — range: [0, 100]
  • Mean of Step Accuracy (1-5) and Process Consistency (1-5) ratings, rescaled to [0, 100]. Step Accuracy measures alignment between textual instructions and subfigures; Process Consistency measures logical inheritance across sequential steps.

Input / output format

Input: Natural-language geometric construction problem description, optionally accompanied by a reference diagram. Prompts are standardized into two templates: one for code generation (LLMs/LRMs) and one for direct visual/textual step generation (UMMs).

Output: For LLMs/LRMs: GeoGebra command syntax code. For UMMs: Natural-language descriptions of drawing steps and/or rendered diagrams.

Scoring recipe

def score_vlm_t(pred_steps, ref_steps):
    # GPT-4o judge evaluates logical completeness, geometric correctness, stepwise coherence
    return judge_score(pred_steps, ref_steps, scale=5)

def score_vlm_i_res(pred_diagram, ref_diagram):
    # GPT-4o judge evaluates geometric consistency and constraint satisfaction
    return judge_score(pred_diagram, ref_diagram, scale=5)

def score_vlm_i_mid(pred_steps, ref_steps):
    step_acc = judge_score(pred_steps, ref_steps, criterion='Step Accuracy', scale=5)
    process_cons = judge_score(pred_steps, ref_steps, criterion='Process Consistency', scale=5)
    return ((step_acc + process_cons) / 2) * 20

Common pitfalls

• Ignoring stylistic variations in favor of logically equivalent strategies that achieve the same geometric objectives.
• Prioritizing perceptual similarity over structural correctness when judging final diagrams.
• Failing to preserve auxiliary traces in ruler-and-compass problems, which breaks the inspectable reasoning chain.
• Using temperature > 0.0 during inference, which introduces stochasticity and violates the deterministic evaluation protocol.

Evidence (verbatim from paper)

The evaluation process compares a model’s generated construction steps against expert-written reference steps, emphasizing logical completeness, geometric correctness, and stepwise coherence. A rubric-based scoring from 1 to 5 is adopted, where higher scores indicate precise and faithful reasoning. To maintain scoring consistency, the evaluator is instructed to ignore stylistic variations and accept logically equivalent strategies that achieve the same geometric objectives. The final score serves as a proxy for the model’s ability to translate visual reasoning tasks into accurate symbolic instructions, and directly contributes to the VLM-T metric reported in [subsection 4.3].

Citation

@misc{wei2025ggbench,
  title={GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2511.11134}
}

• arXiv: 2511.11134