Humaneval X Eval

humaneval-x-eval

CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X — Zheng et al. (2023) (arXiv:2303.17568, 2023)

What this evaluates

Evaluates multilingual code generation and code translation capabilities across five programming languages (C++, Java, JavaScript, Go, Python). It measures functional correctness by executing generated code against a suite of test cases for each problem.

Datasets

• HumanEval-X — total 820; splits: test (820)

Metrics

• pass@k (primary) — range: [0, 1]
  • Estimates the probability that at least one of k samples passes all test cases out of n total generations: pass@k = E[1 - C(n-c, k) / C(n, k)], where c is the number of samples passing all tests, n=200, and k∈{1, 10, 100}. Averaged over all problems.

Input / output format

Input: For code generation: function declaration and docstring. For code translation: function declaration in the target language and canonical solution in the source language.

Output: Generated function implementation in the target language.

Scoring recipe

def compute_pass_at_k(generations, test_cases, k=10, n=200):
    passed = [1 if run_tests(gen, test_cases) else 0 for gen in generations]
    c = sum(passed)
    if n < k or c == n: return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

# Average single-problem pass@k over all problems in the dataset

Common pitfalls

• Using string similarity metrics like BLEU or CodeBLEU instead of functional correctness via test execution.
• Ignoring language-specific behaviors in test cases (e.g., rounding rules, string prefixes) which can cause false negatives if not handled per language.
• Failing to account for budget allocation when evaluating multilingual models across multiple languages simultaneously.

Evidence (verbatim from paper)

Metric. For both tasks, we use test cases to evaluate the exact functional correctness of the generated code, measuring the performance with pass@$k$ (Kulal et al., 2019), making it real-world useful and also completely different from the string similarity metrics like BLEU(Papineni et al., 2002), and CodeBLEU(Ren et al., 2020; Lu et al., 2021; Zhu et al., 2022). Specifically, we use the unbiased method to estimate pass@$k$ (Chen et al., 2021): pass@k:=E[1-(n-c k)/(n k)],n=200,k∈{1,10,100} where n is the total number of generation (n=200 in this work), k is the sampling budget (typically k∈{1, 10, 100}) and c is the number of samples that pass all test cases.

Citation

@misc{zheng2023codegeex,
  title={CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X},
  author={Zheng et al. (2023)},
  year={2023},
  note={arXiv:2303.17568}
}

• arXiv: 2303.17568