Geoqa Eval

geoqa-eval

GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning — Jiaqi Chen et al. (arXiv:2105.14517, 2021)

What this evaluates

Evaluates a model's ability to perform multimodal numerical reasoning on geometric problems by generating executable symbolic programs from text and diagram inputs. The model must fuse cross-modal information to predict step-by-step reasoning programs. These programs are then executed to select the correct multiple-choice answer from the given options.

Datasets

• GeoQA — total 4998; splits: test (-1); repo https://github.com/chen-judge/GeoQA

Metrics

• answer accuracy (primary) — range: percent
  • Percentage of questions where the executed predicted program yields the correct multiple-choice option compared to the ground truth. Calculated as (correct predictions / total questions) * 100.

Input / output format

Input: A geometric problem presented as a text description and a corresponding diagram image.

Output: A sequence of executable symbolic programs, which are executed to produce a final multiple-choice answer.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred_prog, gold_ans in zip(predictions, gold_answers):
        try:
            result = execute_program(pred_prog)
            if result == gold_ans:
                correct += 1
        except ExecutionError:
            pass
    return (correct / len(gold_answers)) * 100

Common pitfalls

• Programs may execute to 'no result' (answer not in options or invalid syntax), which counts as incorrect and significantly lowers accuracy.
• Beam size heavily influences program generation and final accuracy; results must be reported with the specific beam size used (e.g., BS=10 vs BS=100).
• Text-only baselines perform poorly compared to multimodal ones; evaluating without diagram input misrepresents the task's requirements.

Evidence (verbatim from paper)

We conduct experiments on GeoQA dataset, and adopt answer accuracy as the evaluation metric. ... After the searched sequence program is executed, there will be three situations: right answer, wrong answer, and no result.

Citation

@misc{chen2021geoqa,
  title={GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning},
  author={Jiaqi Chen et al.},
  year={2021},
  note={arXiv:2105.14517}
}

• arXiv: 2105.14517