Bigcodebench Eval

bigcodebench-eval

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions — Terry Yue Zhuo et al. (arXiv:2406.15877, 2024)

What this evaluates

Evaluates large language models' ability to generate correct, executable code for complex programming tasks requiring diverse function calls and compositional reasoning. It also probes instruction-following capabilities by comparing performance on verbose prompts versus condensed natural-language instructions.

Datasets

• BigCodeBench — total 1140; splits: test (1140); repo https://github.com/bigcode-project/bigcodebench

Metrics

• Pass@1 (primary) — range: [0, 1]
  • Unbiased Pass@K (Chen et al., 2021) estimates the probability that at least one of K generated samples passes all test cases. Calibrated Pass@1 injects missing imports/constants before execution to account for model laziness.

Input / output format

Input: Complex programming task descriptions requiring specific function calls from various libraries, provided as zero-shot prompts. For the Instruct variant, prompts are condensed natural-language instructions.

Output: Python code snippets intended to solve the given task.

Scoring recipe

def calculate_pass_at_1(generated_code, test_cases):
    calibrated_code = add_missing_setup(generated_code)
    return all(run_test(calibrated_code, tc) for tc in test_cases)
# Aggregate pass rates across all tasks to compute mean Pass@1

Common pitfalls

• Model laziness causes omission of essential imports/constants, requiring calibration to avoid false negatives.
• Computational limits often force N=5 samples, violating the N>=K requirement for unbiased Pass@K estimation.
• Condensed instructions in the Instruct variant introduce ambiguity, lowering scores independent of coding ability.

Evidence (verbatim from paper)

Our evaluation uses the unbiased version of Pass@K (Chen et al., 2021) to accurately assess the functional correctness of generated code snippets by LLMs. ... we calibrate the generation quality by adding the missing setup and calculate Pass@1, which is denoted as calibrated Pass@1.

Citation

@misc{zhuo2024bigcodebench,
  title={BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions},
  author={Terry Yue Zhuo et al.},
  year={2024},
  note={arXiv:2406.15877}
}

• arXiv: 2406.15877