Humaneval Eval

humaneval-eval

StarCoder: may the source be with you! — Raymond Li et al. (2023) (arXiv:2305.06161, 2023)

What this evaluates

Evaluates a model's ability to generate correct, executable Python code from natural language function descriptions and signatures. It measures functional correctness by checking if generated code passes hidden unit tests.

Datasets

• HumanEval — total 164; splits: test (164); HF openai/humaneval

Metrics

• pass@1 (primary) — range: [0, 1]
  • The fraction of benchmark problems for which at least one of k generated samples passes all provided test cases. The paper focuses on pass@1 (k=1), using sampling temperature 0.2 and generating n=200 samples per problem to estimate the metric reliably.

Input / output format

Input: Python function signature, docstring containing a natural language description, and a set of hidden test cases/assertions.

Output: Generated Python code (typically the function body or full function) intended to satisfy the docstring and pass all test cases.

Scoring recipe

def compute_pass_at_1(generated_samples, test_cases):
    passed = 0
    for problem in generated_samples:
        solved = False
        for sample in problem:
            if run_and_check_tests(sample, test_cases):
                solved = True
                break
        if solved:
            passed += 1
    return passed / len(generated_samples)

Common pitfalls

• Using the wrong sampling temperature (0.2 for pass@1 vs 0.8 for pass@k>1) leads to non-comparable scores.
• Failing to execute generated code in a sandboxed environment or using insufficient test cases can inflate scores.
• Not generating enough samples (n=200 recommended) to accurately estimate pass@1, especially for harder problems.

Evidence (verbatim from paper)

We report performance using the pass@k metric (Chen et al., 2021): the total fraction of benchmark problems solved, where a problem is considered solved if any one of k code samples passes every test case. Like Chen et al. (2021), we use sampling temperature 0.2 for pass@1, and temperature 0.8 for k > 1. We generate n = 200 samples for all experiments with open-access models.

Citation

@misc{li2023starcoder,
  title={StarCoder: may the source be with you!},
  author={Raymond Li et al. (2023)},
  year={2023},
  note={arXiv:2305.06161}
}

• arXiv: 2305.06161