Mceval Eval

mceval-eval

McEval: Massively Multilingual Code Evaluation — Chai et al. (2024) (arXiv:2406.07436, 2024)

What this evaluates

Evaluates the multilingual code generation, explanation, and completion capabilities of LLMs across 40 programming languages. It measures how well models can produce correct code, explain code logic, and complete code snippets in diverse syntaxes. The benchmark highlights performance disparities between closed-source and open-source models, particularly in non-Python languages.

Datasets

• MCEVAL — total 16000; splits: test (-1)

Metrics

• Pass@1 (%) (primary) — range: percent
  • Percentage of generated code samples that pass all provided unit tests on the first greedy generation attempt.

Input / output format

Input: Programming language-specific prompts for code generation, code explanation, or code completion tasks.

Output: Generated code, natural language code explanations, or completed code snippets.

Scoring recipe

def compute_pass_at_1(predictions, tests):
    passed = 0
    for pred, test_suite in zip(predictions, tests):
        if run_tests(pred, test_suite):
            passed += 1
    return (passed / len(predictions)) * 100

Common pitfalls

• Data contamination must be strictly avoided; exact matches from the benchmark were explicitly removed from training corpora.
• Performance varies drastically across languages, with non-Python languages often showing significantly lower scores due to training data scarcity.
• Evaluation relies on unit tests which may not capture semantic correctness or edge cases outside the provided test suite.

Evidence (verbatim from paper)

We adopt the greedy Pass@1 (%) metric (Kulal et al., 2019; Chen et al., 2021) for evaluations. For closed-source LLMs, the answers are generated by the official API. Table 1 shows the Pass@1 results of various models on MCEVAL for multilingual code generation task.

Citation

@misc{chai2024mceval,
  title={McEval: Massively Multilingual Code Evaluation},
  author={Chai et al. (2024)},
  year={2024},
  note={arXiv:2406.07436}
}

• arXiv: 2406.07436