Starcoder2 Code Eval

starcoder2-code-eval

StarCoder 2 and The Stack v2: The Next Generation — Lozhkov et al. (2024) (arXiv:2402.19173, 2024)

What this evaluates

Evaluates code generation, completion, and bug-fixing capabilities across multiple programming languages and libraries. It probes a model's ability to write correct functions from prompts, translate code across languages, and fix existing buggy code using standard and enhanced benchmarks.

Datasets

• HumanEval — total 164; splits: test (164)
• MBPP — total 500; splits: test (500)
• EvalPlus — total ?; splits: test (-1); repo https://github.com/evalplus/evalplus
• MultiPL-E — total ?; splits: test (-1)
• DS-1000 — total 1000; splits: test (1000)
• HumanEvalFix — total ?; splits: test (-1)

Metrics

• pass@1 (primary) — range: [0, 1]
  • The probability that at least one of the k=1 generated samples passes all hidden unit tests for a given problem. Reported as mean pass@1 across all problems.

Input / output format

Input: Function signature and docstring (HumanEval/MBPP), natural language description followed by tests (MBPP), or buggy code snippet (HumanEvalFix).

Output: Completed function code or fixed code snippet.

Scoring recipe

def pass_at_1(generated_code, tests):
    for test in tests:
        if not test.run(generated_code):
            return 0.0
    return 1.0

mean_pass_at_1 = sum(pass_at_1(gen, tests) for gen, tests in dataset) / len(dataset)

Common pitfalls

• Using original HumanEval/MBPP instead of EvalPlus variants yields inflated scores due to inadequate hidden tests that miss subtle bugs.
• Base models score poorly on HumanEvalFix unless prompted with specific formats (e.g., 'Issue' vs 'Instruct'); instruction-tuned models are compared separately.
• MultiPL-E and DS-1000 results are averaged over 50 and 40 samples respectively at temperature 0.2 and top-p 0.95, not greedy decoding, which differs from HumanEval/MBPP evaluation settings.

Evidence (verbatim from paper)

Following recent work on Code LLMs (Rozière et al., 2023; Guo et al., 2024), we use greedy decoding and report the mean pass@1 (mean success rate) for all problems in the benchmark. ... For rigorous evaluation, we adopt the EvalPlus framework in this study.

Citation

@misc{lozhkov2024starcoder2,
  title={StarCoder 2 and The Stack v2: The Next Generation},
  author={Lozhkov et al. (2024)},
  year={2024},
  note={arXiv:2402.19173}
}

• arXiv: 2402.19173