Mbpp Eval

mbpp-eval

Program Synthesis with Large Language Models — Austin et al. (2021) (arXiv:2108.07732, 2021)

What this evaluates

Evaluates a model's ability to generate correct, self-contained Python functions from natural language problem descriptions. It probes basic programming logic, standard library usage, and semantic grounding of simple algorithmic tasks.

Datasets

• Mostly Basic Programming Problems (MBPP) — total 974; splits: train (374), val (90), test (500); repo https://github.com/google-research/google-research/tree/master/mbpp

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of test problems for which the generated Python function passes all three provided semantic test cases.

Input / output format

Input: Natural language problem statement (typically one sentence), optionally preceded by few-shot examples.

Output: A self-contained Python function definition that solves the described problem without printing to the console.

Scoring recipe

correct = 0
for problem in test_set:
    code = model.generate(problem.prompt)
    if execute_and_check(code, problem.test_cases):
        correct += 1
return correct / len(test_set)

Common pitfalls

• Original dataset questions may have ambiguous descriptions or non-standard function signatures; the 'edited' subset fixes this.
• Generated code must be self-contained and must not print results to the console to pass the test harness.
• Test cases check semantic correctness, but some original problems had mismatches between the text description and the expected operations.

Evidence (verbatim from paper)

We asked crowd-sourcing participants to write a short problem statement, a single self-contained Python function solving the problem specified, and three test cases that check for semantic correctness of the function. ... In the experiments described later in the paper, we hold out 10 problems for few-shot prompting, another 500 as our test dataset (which is used to evaluate both few-shot inference and fine-tuned models), 374 problems for fine-tuning, and the rest for validation.

Citation

@misc{austin2021programsynthesis,
  title={Program Synthesis with Large Language Models},
  author={Austin et al. (2021)},
  year={2021},
  note={arXiv:2108.07732}
}

• arXiv: 2108.07732