Codemmlu Eval

codemmlu-eval

CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs — Dung Nguyen Manh et al. (2024) (arXiv:2410.01999, 2024)

What this evaluates

Evaluates code understanding and reasoning capabilities of large language models using a multiple-choice question format. It probes syntactic knowledge, semantic comprehension, and real-world software engineering problem-solving, revealing gaps in true code reasoning compared to open-ended generation benchmarks.

Datasets

• CodeMMLU — total 20000; splits: test (-1)

Metrics

• accuracy % (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as the average accuracy across all subject categories in the benchmark.

Input / output format

Input: Multiple-choice questions containing code snippets or software engineering problems with four answer options (A, B, C, D).

Output: A single letter (A, B, C, or D) or a phrase containing the pattern "answer is A|B|C|D".

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        match = re.search(r'\b([A-D])\b', pred)
        if not match:
            match = re.search(r'answer is\s*([A-D])', pred, re.IGNORECASE)
        if match and match.group(1).upper() == gold_ans.upper():
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Applying Chain-of-Thought (CoT) prompting often degrades performance on knowledge-seeking code tasks rather than improving it.
• Model scores are highly sensitive to the permutation of answer choices (A/B/C/D), indicating strong selection bias in the MCQ format.
• High performance on open-ended code generation benchmarks (e.g., HumanEval) does not reliably predict accuracy on multiple-choice comprehension tasks.

Evidence (verbatim from paper)

The evaluation results (accuracy %) of different language models across the CodeMMLU task (CodeMMLU column represents the accuracy average among all subject). In order to maintain this advantage, we only apply simple regex methods to extract the selection answer (i.e., extract by directly answering (A|B|C|D) or containing the pattern "answer is A|B|C|D"). The model response is required to be parsable; otherwise, it will be marked as unanswered.

Citation

@misc{nguyenmanh2024codemmlu,
  title={CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs},
  author={Dung Nguyen Manh et al. (2024)},
  year={2024},
  note={arXiv:2410.01999}
}

• arXiv: 2410.01999