Mathcoder Eval

mathcoder-eval

MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning — Wang et al. (2023) (arXiv:2310.03731, 2023)

What this evaluates

Evaluates large language models' ability to solve mathematical word problems across varying difficulty levels and subjects, including elementary, high school, and collegiate mathematics. It specifically probes the model's capacity for code-interleaved reasoning and execution-aware autoregression.

Datasets

• GSM8K — total ?; splits: test (-1)
• MATH — total ?; splits: test (-1)
• SVAMP — total ?; splits: test (-1)
• Mathematics — total ?; splits: test (-1)
• SimulEq — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Exact-match accuracy: the percentage of problems where the model's predicted final answer exactly matches the ground truth answer.

Input / output format

Input: Natural language mathematical word problems.

Output: A final numerical or symbolic answer, optionally preceded by reasoning steps and code execution blocks.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Baselines are evaluated with CoT or few-shot prompting to maximize performance, while MathCoder is strictly zero-shot; comparing them directly without noting the prompt difference is misleading.
• The paper distinguishes between models that predict execution results internally versus those that use an external Python interpreter during inference; results vary significantly between these two settings.

Evidence (verbatim from paper)

Tab. 5 (#3) improves the accuracy across four out of five datasets, resulting in a rise in the average accuracy from 69.1% to 70.2%, which aligns with the hypothesis that by computing the loss only on natural language and code, the model can focus more on the math problem-solving skills itself, thus making the supervised fine-tuning more effective.

Citation

@misc{wang2023mathcoder,
  title={MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2310.03731}
}

• arXiv: 2310.03731