Codereviewqa Eval

codereviewqa-eval

CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models — Lin et al. (2025) (arXiv:2503.16167, 2025)

What this evaluates

This benchmark probes large language models' ability to comprehend implicit code review intent by decomposing the task into change type recognition, change localization, and solution identification. It uses multiple-choice questions to evaluate whether models can accurately interpret pre-change code and reviewer comments without relying on surface-level code generation.

Datasets

• CodeReviewQA — total 900; splits: test (900)

Metrics

• accuracy (primary) — range: [0, 1]
  • Models are evaluated across all permutations of answer options (N! runs). A question is counted as correct only if the model selects the correct option in all N! runs. The final score is the fraction of questions answered correctly across all permutations.

Input / output format

Input: Multiple-choice prompt containing a task definition, a question presenting a code review scenario (with programming language markers, pre-change code H_pre, and natural language review R_nl), and a list of alphabetically labeled options (A, B, C, D).

Output: A single letter symbol (A, B, C, or D) corresponding to the selected option.

Scoring recipe

correct = 0
for q in dataset:
    perms = all_permutations(q.options)
    all_match = True
    for p in perms:
        prompt = build_prompt(q, p)
        pred = extract_top_logit_symbol(prompt)
        if pred != q.correct_symbol:
            all_match = False
            break
    if all_match:
        correct += 1
return correct / len(dataset)

Common pitfalls

• Models may guess correctly on a single option ordering but fail on others; the protocol requires correctness across ALL N! permutations.
• Answer extraction uses max log-probability of the symbol token rather than normalized sequence likelihood, making results sensitive to prompt formatting and decoding settings.
• The benchmark measures comprehension via MCQA, so standard code generation metrics (e.g., pass@k) are inappropriate and will not reflect model performance.

Evidence (verbatim from paper)

To be counted as correctly answering that question, the models must select the correct answer for all N! runs, which is a more reliable indicator of the models' understanding (Wang et al., 2025).

Citation

@misc{lin2025codereviewqa,
  title={CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models},
  author={Lin et al. (2025)},
  year={2025},
  note={arXiv:2503.16167}
}

• arXiv: 2503.16167