Cruxeval Eval

cruxeval-eval

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution — Gu et al. (2024) (arXiv:2401.03065, 2024)

What this evaluates

Evaluates a model's ability to reason about and execute short Python functions by predicting outputs given inputs (CRUXEval-I) and predicting inputs given outputs (CRUXEval-O). It probes fundamental code execution and understanding capabilities beyond simple code generation.

Datasets

• CRUXEval — total 800; splits: test (-1)

Metrics

• pass@1 (primary) — range: [0, 1]
  • The fraction of problems where at least one of the N generated samples matches the gold output. Evaluated with generation temperature T=0.2.
• pass@5 — range: [0, 1]
  • The fraction of problems where at least one of the N generated samples matches the gold output. Evaluated with generation temperature T=0.8.

Input / output format

Input: A short Python function (3–13 lines) with a signature and docstring, plus either input arguments (for input-output prediction) or expected output (for output-input prediction).

Output: The predicted output value(s) or input arguments that satisfy the function's specification.

Scoring recipe

def compute_pass_at_k(gold_outputs, generated_samples, k):
    correct = sum(1 for s in generated_samples if s in gold_outputs)
    return correct / len(gold_outputs)

# Per problem:
# N = 100 for non-GPT models, N = 10 for GPT models
# pass@1 uses T=0.2, pass@5 uses T=0.8
# Final score = mean(compute_pass_at_k(gold, samples, k) over all problems)

Common pitfalls

• Temperature settings differ between pass@1 (T=0.2) and pass@5 (T=0.8); using a single temperature for both will invalidate the reported scores.
• Statistical significance is assessed via paired bootstrap over 10,000 samples rather than standard t-tests; replicating confidence intervals requires resampling the exact dataset splits.
• The benchmark uses a generate-and-filter pipeline; models must be evaluated on the original 800 functions, not on filtered or modified subsets.

Evidence (verbatim from paper)

We report both pass@1 scores $(T = 0.2)$ and pass@5 scores $(T = 0.8)$. The results are shown in Fig. 2, and raw scores are provided in the Appendix in Table 2.

Citation

@misc{gu2024cruxeval,
  title={CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution},
  author={Gu et al. (2024)},
  year={2024},
  note={arXiv:2401.03065}
}

• arXiv: 2401.03065