Cruxevalx Eval

cruxevalx-eval

CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution — Xu et al. (2024) (arXiv:2408.13001, 2024)

What this evaluates

This benchmark evaluates large language models' ability to perform bidirectional code reasoning across 19 programming languages. It probes whether models can predict missing inputs given outputs, and predict missing outputs given inputs, testing their understanding of code semantics and execution flow.

Datasets

• CRUXEval-X — total 19000; splits: test (-1)

Metrics

• Pass@1 (primary) — range: percent
  • Pass@1 measures the fraction of problems for which the single generated solution (via greedy decoding) passes the provided test case. It is calculated as the number of passed problems divided by the total number of problems.

Input / output format

Input: For each instance, the model receives a code snippet along with either the expected output (for input reasoning) or the expected input (for output reasoning) from a test case.

Output: The model must generate the missing component: either the input that produces the given output, or the output that results from the given input.

Scoring recipe

def compute_pass_at_1(predictions, test_cases):
    passed = 0
    for pred, test in zip(predictions, test_cases):
        # Execute generated code with test input/output
        if execute_and_check(pred, test):
            passed += 1
    return (passed / len(predictions)) * 100

Common pitfalls

• Evaluation uses greedy decoding (temperature=0), so Pass@1 here is equivalent to a single-run pass rate rather than a sampling-based estimate.
• The benchmark spans 19 languages with both dynamic and static type systems; models may struggle with type inference or syntax differences despite cross-lingual generalization.
• Test cases are designed with exponential search spaces to prevent memorization, so models must genuinely reason about code execution rather than recall answers.

Evidence (verbatim from paper)

We use pass@1 (Kulal et al., [2019]; Chen et al., [2021]) to evaluate both tasks. We set the temperature to 0 and employ greedy decoding for generation as prior work (Cao et al., [2024a]).

Citation

@misc{xu2024cruxevalx,
  title={CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution},
  author={Xu et al. (2024)},
  year={2024},
  note={arXiv:2408.13001}
}

• arXiv: 2408.13001