Humanevalfix Eval

humanevalfix-eval

OctoPack: Instruction Tuning Code Large Language Models — Muennighoff et al. (2023) (arXiv:2308.07124, 2023)

What this evaluates

Evaluates a model's ability to debug and fix buggy code by generating corrected implementations that pass provided unit tests. It probes code repair capabilities across multiple programming languages.

Datasets

• HumanEvalFix — total ?; splits: test (-1)

Metrics

• pass rate (primary) — range: percent
  • Percentage of generated code fixes that pass all provided unit tests for a given problem. Calculated as (number of passing fixes / total problems) * 100.

Input / output format

Input: A buggy function signature and body, accompanied by its unit tests. Optionally, docstrings may be provided as ground truth hints.

Output: The corrected function implementation (code only).

Scoring recipe

def score(predictions, gold_tests):
    passed = 0
    for pred_code, tests in zip(predictions, gold_tests):
        if run_tests(pred_code, tests): # All unit tests pass
            passed += 1
    return (passed / len(predictions)) * 100

Common pitfalls

• Models may ignore the buggy function and generate entirely new code from scratch if docstrings are provided, bypassing the actual repair task.
• Prompting format significantly impacts performance; commit-style prompts may outperform standard instruction prompts depending on pretraining data.

Evidence (verbatim from paper)

The default version of HUMANEVALFIX does not include docstrings, but only provides the unit tests to the model alongside the buggy function. Solving from docstrings is generally easier for models than from tests, as models can also solve it via pure code synthesis without looking at the buggy function at all.

Citation

@misc{muennighoff2023octopack,
  title={OctoPack: Instruction Tuning Code Large Language Models},
  author={Muennighoff et al. (2023)},
  year={2023},
  note={arXiv:2308.07124}
}

• arXiv: 2308.07124