Xcodeeval Eval

xcodeeval-eval

xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval — Khan et al. (2023) (arXiv:2303.03004, 2023)

What this evaluates

Evaluates large language models on multilingual code understanding, generation, translation, and retrieval across 11 programming languages. It probes the model's ability to produce executable, correct code by validating outputs against unit tests rather than relying on lexical overlap.

Datasets

• xCodeEval — total ?; splits: train (-1), validation (-1), test (-1); repo https://github.com/ntunlp/xCodeEval

Metrics

• pass@5 (primary) — range: [0, 1]
  • The probability that at least one of k=5 generated solutions passes all provided unit tests. Computed as 1 - (C(n-c, k) / C(n, k)) where n is total samples and c is correct samples.
• macro-F1 — range: [0, 1]
  • Macro-averaged F1 score for multi-label tag classification tasks, computed per label and averaged.
• accuracy — range: [0, 1]
  • Standard accuracy for binary code compilation classification (compileable vs not).

Input / output format

Input: Natural language problem description, sample input-output pairs, and/or buggy/source code depending on the task (e.g., Program Synthesis, APR, Code Translation, Tag Classification).

Output: Generated executable code in the target programming language.

Scoring recipe

def compute_pass_at_k(generations, unit_tests, k=5):
    passed_count = 0
    for gen in generations[:k]:
        result = execute_code(gen, unit_tests)
        if result == 'PASSED':
            passed_count += 1
            break
    return passed_count / k
# execute_code compiles/runs the code, checks time/memory limits, and verifies output against all unit tests.
# Fails on compilation error, runtime error, memory/time limit exceeded, or wrong answer on any single test.

Common pitfalls

• Models may achieve high lexical overlap or static analysis scores but fail execution-based validation; the benchmark explicitly requires actual program correctness.
• Failing on a single unit test flags the entire program as buggy, so partial correctness does not count toward pass@k.
• Execution requires handling 44 compiler/interpreter versions and strict time/memory limits, which can cause false negatives if the evaluation environment is not configured identically to the benchmark.

Evidence (verbatim from paper)

All of our proposed generative tasks are evaluated with execution-based unit tests by ExecEval. The program will be flagged as buggy (i-v) even when it fails on a single unit test. TABLE 3 - Performance of gpt-3.5-turbo on XCODEVAL : ... Program Synthesis (T) | pass@5 | 25.37 | 30.59 | 31.36 | 31.03 | 29.74 | 22.74 | 26.87 | 30.17 | 33.98 | 33.72 | 10.28 | 27.8

Citation

@misc{khan2023xcodeeval,
  title={xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval},
  author={Khan et al. (2023)},
  year={2023},
  note={arXiv:2303.03004}
}

• arXiv: 2303.03004