Comat Eval

comat-eval

CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning — Joshua Ong Jun Leang, Aryo Pradipta Gema, Shay B. Cohen (2024) (arXiv:2410.10336, 2024)

What this evaluates

Evaluates large language models on mathematical reasoning across diverse difficulty levels and languages. It probes the model's ability to convert natural language word problems into structured symbolic representations and execute step-by-step logical derivations without external solvers.

Datasets

• AQUA — total ?; splits: test (-1)
• MultiArith — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• MMLU-Redux — total ?; splits: test (-1)
• Olympiad Bench (English) — total ?; splits: test (-1)
• GaoKao — total ?; splits: test (-1)
• Olympiad Bench (Chinese) — total ?; splits: test (-1)

Metrics

• exact match (primary) — range: percent
  • Accuracy calculated as the percentage of instances where the model's predicted answer fully matches the ground truth answer. For multiple-choice questions, the full answer text must match, not just the option letter.
• GPT-4o-mini alignment — range: percent
  • Automated evaluation using GPT-4o-mini to judge whether the model's predicted answer aligns with the ground truth for Olympiad Bench problems.

Input / output format

Input: Natural language mathematical word problems, presented as multiple-choice questions or open-ended string-valued questions.

Output: A complete predicted answer string. For multiple-choice datasets, the full answer text (not just the option code like A, B, C, or D).

Scoring recipe

def compute_metrics(predictions, golds, dataset_name):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if dataset_name in ['AQUA', 'MMLU-Redux', 'GaoKao', 'MultiArith', 'GSM8K']:
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
        elif dataset_name in ['Olympiad Bench (English)', 'Olympiad Bench (Chinese)']:
            if judge_with_gpt4o_mini(pred, gold):
                correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Evaluating multiple-choice answers by matching only the option code (e.g., 'A') instead of the full answer text, which the paper explicitly warns against for datasets like MMLU-Redux.
• Relying on external verification solvers (e.g., Datalog, Z3, LEAN) during the reasoning execution phase, which are excluded from the CoMAT protocol to ensure a focused LLM-only comparison.
• Using GPT-4o for generating Faithful CoT baselines, as it tends to produce more invalid outputs compared to GPT-4, which the authors specifically note.

Evidence (verbatim from paper)

For multiple-choice datasets, we use exact match metrics, requiring the predicted answer to fully match the correct one, not just the option code (e.g., A, B). This ensures evaluation based on complete responses, addressing clarity concerns in datasets like MMLU-Redux. For string-valued answers, we also use an exact match in GSM8K. On Olympiad Bench, we use GPT-4o-mini as a benchmark to evaluate how well the model’s answers align with the ground truth.

Citation

@misc{ong2024comat,
  title={CoMAT: Chain of Mathematically Annotated Thought Improves Mathematical Reasoning},
  author={Joshua Ong Jun Leang, Aryo Pradipta Gema, Shay B. Cohen (2024)},
  year={2024},
  note={arXiv:2410.10336}
}

• arXiv: 2410.10336