Code Generation Eval

code-generation-eval

Code Llama: Open Foundation Models for Code — Rozière et al. (2023) (arXiv:2308.12950, 2023)

What this evaluates

Evaluates a model's capability to generate syntactically correct and functionally executable code from natural language specifications or prompts across multiple programming languages and difficulty levels.

Datasets

• HumanEval — total ?; splits: test (-1)
• MBPP — total ?; splits: test (-1)
• APPS — total ?; splits: test (-1)
• MultiPL-E — total ?; splits: test (-1)

Metrics

• pass@k (primary) — range: percent
  • Estimates the probability that at least one of k generated solutions passes all provided unit tests. Computed per problem as 1 - (C(n-c, k) / C(n, k)) where n is total samples and c is correct samples, then averaged across problems.
• exact match — range: percent
  • Binary metric checking if the generated code string exactly matches the ground truth reference solution.
• perplexity — range: other
  • Exponential of the average negative log-likelihood of the tokens in the sequence, used to measure language modeling quality on long contexts.

Input / output format

Input: Natural language problem description or prompt, optionally with few-shot examples (e.g., two-shot for APPS). For infilling tasks, code with masked spans in prefix-suffix-middle or suffix-prefix-middle format.

Output: A complete Python code snippet or function body intended to solve the problem, or the masked code segment for infilling tasks.

Scoring recipe

def pass_at_k(n, c, k):
    if n - c < k: return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)

scores = []
for problem in dataset:
    samples = generate(problem.prompt, k=100)
    correct = sum(1 for s in samples if run_tests(s, problem.tests))
    scores.append(pass_at_k(100, correct, k))
return sum(scores) / len(scores) * 100

Common pitfalls

• Sampling parameters differ across benchmarks (e.g., temperature 0.1 for HumanEval/MBPP vs. p=0.95 and temp=0.6 for APPS), making direct comparisons sensitive to generation settings.
• APPS results are reported with raw predictions without filtering by test cases from the prompt, which may overestimate real-world correctness.
• Random span infilling in SPM format suffers from token healing issues, artificially lowering scores if the model cannot reconstruct exact character sequences.

Evidence (verbatim from paper)

The value of model specialization. We observe that model specialization is yields a boost in code generation capabilities when comparing Llama 2 to CODE Llama and CODE Llama to CODE Llama - Python. Llama 2 was trained on 2T tokens, and training on only 500B of extra tokens from a code-heavy dataset results in massive performance gains on both HumanEval and MBPP, to the point that Llama 2 70B is roughly equivalent to CODE Llama 7B on Python coding benchmarks. Although CODE Llama was trained on more than two epochs of our code dataset, which contains our entire Python dataset, training on 100B extra tokens of a Python-heavy data mix leads to significant gains on Python code generation benchmarks, between 4.3% points and 8.3% points in HumanEval pass@1 and between 1.2% points and 6.4% points in MBPP pass@1.

Citation

@misc{roziere2023codellama,
  title={Code Llama: Open Foundation Models for Code},
  author={Rozière et al. (2023)},
  year={2023},
  note={arXiv:2308.12950}
}

• arXiv: 2308.12950