Mathodyssey Eval

mathodyssey-eval

MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data — Fang et al. (2024) (arXiv:2406.18321, 2024)

What this evaluates

Evaluates large language models' mathematical reasoning capabilities across Olympiad, high school, and university-level problems. It probes multi-step logic, chain-of-thought reasoning, and advanced derivations in algebra, calculus, and number theory.

Datasets

• MathOdyssey — total 387; splits: test (387)

Metrics

• accuracy (primary) — range: [0, 1]
  • Accuracy is the fraction of correctly answered problems out of the total. Correctness is determined by verifying mathematical equivalence between the predicted and ground-truth answers using symbolic computation, numerical tolerance (correct to at least two decimal places), and multiple-choice matching rules.

Input / output format

Input: Mathematical problem statement (text), evaluated using chain-of-thought prompting.

Output: Step-by-step reasoning followed by a final answer (exact value, choice label, or numerical approximation).

Scoring recipe

def is_correct(pred, gold):
    if pred == gold: return True
    if is_numerically_equivalent(pred, gold, tol=0.01): return True
    if is_symbolically_equivalent(pred, gold): return True
    if is_multiple_choice_match(pred, gold): return True
    return False

accuracy = sum(1 for p, g in zip(predictions, golds) if is_correct(p, g)) / len(golds)

Common pitfalls

• Assuming exact string matching is sufficient; the protocol explicitly requires symbolic and numerical equivalence checking.
• Overlooking the chain-of-thought requirement; models must be prompted with CoT to generate step-by-step reasoning before the final answer.
• Misinterpreting numerical tolerance; answers must be correct to at least two decimal places, not just rounded integers.

Evidence (verbatim from paper)

We use GPT-4 to assist in evaluating model accuracy, particularly for open-answer questions. The metric measures the similarity between the predicted and ground truth answers. In the MathOdyssey dataset, various types of questions and answers are included. We employ a prompt-based method to provide scores for evaluation, considering the following criteria: Mathematical Equivalence: Verify answers based on mathematical equivalence using advanced tools like symbolic computation software to confirm the equivalence of different algebraic or symbolic expressions. Scoring: Assign a score of ‘1’ for answers that match or are equivalent to the provided solution (exact value, choice label, or correctly rounded numerical approximation). Assign a score of ‘0’ for incorrect answers without providing explanatory feedback.

Citation

@misc{fang2024mathodyssey,
  title={MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Data},
  author={Fang et al. (2024)},
  year={2024},
  note={arXiv:2406.18321}
}

• arXiv: 2406.18321