Ioaa LLM Eval

ioaa-llm-eval

Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA) — Lucas Carrit Delgado Pinheiro et al. (2025) (arXiv:2510.05016, 2025)

What this evaluates

Evaluates LLMs' ability to solve advanced astronomy and astrophysics problems, focusing on geometric/spatial reasoning, physical calculations, and multimodal data analysis. It benchmarks performance against human Olympiad participants using official scoring rubrics.

Datasets

• IOAA (International Olympiad on Astronomy and Astrophysics) — total ?; splits: test (-1); repo https://github.com/OSU-NLP-Group/LLM-IOAA

Metrics

• score (primary) — range: percent
  • Percentage of total available points earned by the model, calculated using official IOAA grading rubrics. Scores are normalized to percentages across theory and data analysis exams.
• medal_threshold — range: other
  • Categorical ranking (bronze, silver, gold) based on the model's score relative to the human median score: bronze for 100–130%, silver for 130–160%, and gold for >160% of the median.
• category_accuracy — range: percent
  • Percentage score broken down by problem type: Category I (Geometric/Spatial) and Category II (Physics/Mathematics).

Input / output format

Input: Text-based astronomy/astrophysics exam questions, supplemented with images, plots, and data tables for data analysis sections.

Output: Model-generated step-by-step solutions, calculations, and final answers to each exam problem.

Scoring recipe

def compute_score(predictions, rubrics):
    total_points = 0
    earned_points = 0
    for q in predictions:
        max_pts = rubrics[q.id].max_points
        earned_pts = rubrics[q.id].grade(q.answer)
        total_points += max_pts
        earned_points += earned_pts
    return (earned_points / total_points) * 100

Common pitfalls

• Small sample size per difficulty level (e.g., 10 easy, 11 medium questions) causes high variance in category-specific scores.
• Data contamination risk: some past exam questions may overlap with known literature or training data (e.g., a 2022 question based on a 1981 study).
• Models struggle with spatial/geometric reasoning even when problems are described purely in text, indicating a fundamental limitation beyond multimodal input processing.

Evidence (verbatim from paper)

All scores are normalized to percentages. Each model is ranked separately with respect to the students.

Citation

@misc{pinheiro2025llmioaa,
  title={Large Language Models Achieve Gold Medal Performance at the International Olympiad on Astronomy & Astrophysics (IOAA)},
  author={Lucas Carrit Delgado Pinheiro et al. (2025)},
  year={2025},
  note={arXiv:2510.05016}
}

• arXiv: 2510.05016