Themis Coderewardbench Eval

themis-coderewardbench-eval

Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring — Paul et al. (2026) (arXiv:2605.00754, 2026)

What this evaluates

Evaluates code reward models on their ability to rank code pairs across five quality dimensions (correctness, efficiency, security, readability, maintainability) and eight programming languages. It probes whether models can generalize beyond functional correctness to assess non-functional code attributes and cross-lingual code preferences.

Datasets

• Themis-CodeRewardBench — total ?; splits: test (-1); repo https://github.com/iNeil77/Themis

Metrics

• preference accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted preferred code pairs out of the total number of preference instances. Calculated as (number of correct predictions) / (total number of pairs).

Input / output format

Input: A prompt/instruction and a pair of code responses representing two alternatives (e.g., buggy vs. fixed, slow vs. fast, insecure vs. secure) for a specific programming language and evaluation criterion.

Output: A binary preference label or score indicating which of the two code responses is preferred according to the specified criterion.

Scoring recipe

correct = 0
total = 0
for prompt, code_a, code_b, gold_label in dataset:
    score_a = model(prompt, code_a)
    score_b = model(prompt, code_b)
    pred = 'a' if score_a > score_b else 'b'
    if pred == gold_label:
        correct += 1
    total += 1
return correct / total

Common pitfalls

• Models heavily bias towards functional correctness, often failing to rank non-functional improvements (e.g., security, readability) correctly.
• Performance drops significantly on low-resource languages due to uneven dataset distribution across the eight target languages.
• Synthetically generated inverse instructions may not perfectly align with the original commit intent, introducing noise into preference labels.

Evidence (verbatim from paper)

Deferring to prior findings*(Frick et al., 2025)* and the strong precedent in existing RM evaluation, we use preference accuracy as the evaluation metric on Themis-CodeRewardBench.

Citation

@misc{paul2026themis,
  title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},
  author={Paul et al. (2026)},
  year={2026},
  note={arXiv:2605.00754}
}

• arXiv: 2605.00754