Mathqa Python Eval

mathqa-python-eval

Program Synthesis with Large Language Models — Austin et al. (2021) (arXiv:2108.07732, 2021)

What this evaluates

Evaluates a model's ability to translate complex mathematical word problems into executable Python code that computes a specific numerical answer. It probes semantic grounding of natural language, arithmetic reasoning, and straight-line code generation.

Datasets

• MathQA-Python — total 23914; splits: train (19209), val (2822), test (1883); repo https://github.com/google/trax/blob/master/trax/examples/MathQA_Python_generation_notebook.ipynb

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of test problems for which the generated Python code executes and evaluates to the declared numerical ground-truth answer.

Input / output format

Input: Mathematical word problem, optionally preceded by few-shot examples.

Output: Python code (or DSL code) that computes the ground truth numerical answer.

Scoring recipe

correct = 0
for problem in test_set:
    code = model.generate(problem.prompt)
    if execute_and_check(code, problem.numerical_answer):
        correct += 1
return correct / len(test_set)

Common pitfalls

• Dataset was filtered to 55% of original size to keep only problems where code evaluates to the declared numerical answer.
• Contains mostly straight-line code but complex natural language descriptions, unlike MBPP's control flow heavy tasks.
• Model must return code that computes the exact numerical answer, not just a string.

Evidence (verbatim from paper)

To evaluate whether pre-training on source code is useful for this task, we translate this dataset into a Python program synthesis dataset by translating the ground-truth programs from the domain-specific language given in the paper to Python code. ... We execute the sampled code to check for semantic correctness. This method of checking correctness forced us to filter the MathQA dataset to keep only those problems for which the code evaluates to the declared numerical answer, resulting in us removing 45% of problems. After this filtration we are left with 23914 problems, of which we use 19209 for training, 2822 for validation and 1883 for testing.

Citation

@misc{austin2021programsynthesis,
  title={Program Synthesis with Large Language Models},
  author={Austin et al. (2021)},
  year={2021},
  note={arXiv:2108.07732}
}

• arXiv: 2108.07732