Combigraph Vis Eval

combigraph-vis-eval

CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning — Mahdavi et al. (2025) (arXiv:2510.27094, 2025)

What this evaluates

Evaluates multimodal discrete mathematical reasoning, specifically the ability to parse and solve combinatorial problems involving graphs, grids, and geometric diagrams. It also probes susceptibility to deliberately crafted distractors in multiple-choice formats versus genuine solution construction.

Datasets

• CombiGraph-Vis — total 1135; splits: test (1135); repo https://github.com/combigraphviz2025/combigraph-viz

Metrics

• avg@8 (primary) — range: [0, 1]
  • Mean correctness over 8 independent samples per problem. Calculated as the average of binary correctness scores across all generated samples for each instance, then averaged over the dataset.

Input / output format

Input: A discrete mathematics problem statement, optionally accompanied by an image (e.g., graphs, grids, geometric diagrams), presented in either multiple-choice or yes/no format.

Output: A final answer, either a selected multiple-choice option or a yes/no response.

Scoring recipe

def compute_avg8(predictions_per_problem, gold_answers):
    correct_counts = []
    for preds, gold in zip(predictions_per_problem, gold_answers):
        correct = sum(1 for p in preds if p == gold)
        correct_counts.append(correct / 8)
    return sum(correct_counts) / len(correct_counts)

Common pitfalls

• Models frequently select distractors in multiple-choice questions that are deliberately crafted to appear correct, inflating choice-matching rates without reflecting true reasoning.
• Performance drops significantly on image-tagged problems compared to text-only ones, so evaluating only text versions underestimates the benchmark's difficulty and modality gap.

Evidence (verbatim from paper)

For each model we compute: (i) Standalone avg@8 = mean correctness over 8 samples; and (ii) Among-Choices avg@8 = mean fraction of standalone problems whose final answer lies among the original options (not necessarily correct).

Citation

@misc{mahdavi2025combigraphvis,
  title={CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning},
  author={Mahdavi et al. (2025)},
  year={2025},
  note={arXiv:2510.27094}
}

• arXiv: 2510.27094