Olympiadbench Eval

olympiadbench-eval

OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems — He et al. (2024) (arXiv:2402.14008, 2024)

What this evaluates

This benchmark evaluates the scientific reasoning capabilities of large multimodal models (LMMs) and large language models (LLMs) on Olympiad-level mathematics and physics problems. It specifically probes bilingual (English and Chinese) text-and-image problem solving, computational correctness, and logical consistency in complex, expert-annotated scenarios.

Datasets

• OlympiadBench — total 8476; splits: test (8476); repo https://github.com/OpenBMB/OlympiadBench

Metrics

• micro-average accuracy (primary) — range: percent
  • Calculated as the total number of correctly answered questions divided by the total number of questions across all subsets, reported as a percentage. Automatic scoring is applied to open-ended problems with predefined answer types, while theorem proving problems require manual verification.

Input / output format

Input: Zero-shot prompt template containing the problem text (in English or Chinese), subject category, answer type, and a flag for multiple answers. Multimodal instances include associated images. Prompts explicitly prescribe the expected answer format to facilitate machine scoring.

Output: A final answer or solution string in the explicitly prescribed format specified in the prompt template, designed for machine-readable extraction and automatic scoring.

Scoring recipe

correct_count = 0
total_count = 0
for instance in dataset:
    prediction = model.generate(instance.prompt)
    is_correct = automatic_scoring_pipeline(prediction, instance.gold_answer)
    if is_correct:
        correct_count += 1
    total_count += 1
micro_avg_accuracy = (correct_count / total_count) * 100

Common pitfalls

• Theorem proving problems lack a reliable automatic evaluation method, necessitating manual sampling checks which can introduce subjectivity or sampling bias.
• Some closed-source models return missing responses due to API input length limits or strict formatting requirements, which must be accounted for when calculating averages.
• Models restricted to single-image input will underperform on problems requiring multiple images, skewing modality-specific results if not explicitly flagged.

Evidence (verbatim from paper)

We first apply each model to generate answers for questions in OlympiadBench using prompts formed by prompt template, with open-source models running on NVIDIA A800 GPUs. Then, we run the automatic scoring pipeline to judge the correctness of the answers as described in subsection[3.4]. Finally, we calculate the micro-average accuracy as the comparing metric.

Citation

@misc{he2024olympiadbench,
  title={OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems},
  author={He et al. (2024)},
  year={2024},
  note={arXiv:2402.14008}
}

• arXiv: 2402.14008