rubric-reward-eval
Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling — Sanders et al. (2026) (arXiv:2602.06795, 2026)
What this evaluates
Evaluates whether data-driven reasoning rubrics improve LLM-based trace correctness classification and serve as effective reward signals for reinforcement learning compared to standard LLM judges and verifiable rewards.
Datasets
Metrics
Specificity — range: [0, 1]
- TN / (FP + TN). The percentage of incorrect traces correctly classified as incorrect.
Balanced Accuracy (primary) — range: [0, 1]
- (TN / (2 * (FP + TN))) + (TP / (2 * (TN + TP))). The average of specificity and recall.
F0.5 — range: [0, 1]
- (1 + 0.5^2) * (P * R) / (0.5^2 * P + R). Balanced precision and recall with additional weight on precision.
Patch Completion Rate — range: [0, 1]
- Count of non-empty patches that pass execution tests divided by total non-empty patches.
Input / output format
Input: Problem statement, model-generated reasoning trace, and optionally ground truth answer or domain-specific rubric items.
Output: Binary correctness label (0 or 1) for trace classification, or final answer/patch for downstream evaluation.
Scoring recipe
def compute_metrics(tp, fp, tn, fn):
specificity = tn / (fp + tn)
balanced_acc = (tn / (2 * (fp + tn))) + (tp / (2 * (tn + tp)))
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f05 = (1 + 0.5**2) * (precision * recall) / (0.5**2 * precision + recall)
return specificity, balanced_acc, f05
Common pitfalls
- Long reasoning traces are truncated to fit model context windows, which may alter correctness classification.
- Intermediate reasoning errors do not always lead to incorrect final answers, creating a tradeoff between specificity and recall.
- SWE-Bench unit tests are computationally expensive, preventing full benchmark evaluation during RL training loops.
Evidence (verbatim from paper)
For experiments assessing rubric quality as an artifact for trace classification, we consider the following metrics: Specificity, or $\frac{TN}{FP+TN}$ (the percentage of incorrect traces classified as incorrect by the LLM), balanced accuracy, or $\frac{TN}{2(FP+TN)}+\frac{TP}{2(TN+TP)}$ (the average of the specificity and recall), and F0.5, $(1+\beta^{2})\frac{PR}{\beta^{2}P+R}$ where $\beta=0.5$, $P=\frac{TP}{TP+FP}$, and $R$ is recall.
Citation
@misc{sanders2026generating,
title={Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling},
author={Sanders et al. (2026)},
year={2026},
note={arXiv:2602.06795}
}
1---2name: rubric-reward-eval3description: Evaluates whether data-driven reasoning rubrics improve LLM-based trace correctness classification and serve as effective reward signals for reinforcement learning compared to standard LLM judges and verifiable rewards. Use when the user wants to benchmark on SWE-Bench, NuminaMath, NaturalReasoning, or asks about evaluating this task. Reports Balanced Accuracy.4---56# rubric-reward-eval78> Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling — Sanders et al. (2026) (arXiv:2602.06795, 2026)910## What this evaluates1112Evaluates whether data-driven reasoning rubrics improve LLM-based trace correctness classification and serve as effective reward signals for reinforcement learning compared to standard LLM judges and verifiable rewards.1314## Datasets1516- **SWE-Bench** — total ?; splits: train (1400), test (80); repo https://github.com/SWE-bench/experiments/tree/main/evaluation/verified/20241022_tools_claude-3-5-haiku17- **NuminaMath** — total ?; splits: train (1400), val (340); HF `open-r1/OpenR1-Math-220k`18- **NaturalReasoning** — total ?; splits: train (-1), val (-1); HF `RJT1990/GeneralThoughtArchive`1920## Metrics2122- `Specificity` — range: [0, 1]23 - TN / (FP + TN). The percentage of incorrect traces correctly classified as incorrect.24- `Balanced Accuracy` **(primary)** — range: [0, 1]25 - (TN / (2 * (FP + TN))) + (TP / (2 * (TN + TP))). The average of specificity and recall.26- `F0.5` — range: [0, 1]27 - (1 + 0.5^2) * (P * R) / (0.5^2 * P + R). Balanced precision and recall with additional weight on precision.28- `Patch Completion Rate` — range: [0, 1]29 - Count of non-empty patches that pass execution tests divided by total non-empty patches.3031## Input / output format3233**Input**: Problem statement, model-generated reasoning trace, and optionally ground truth answer or domain-specific rubric items.3435**Output**: Binary correctness label (0 or 1) for trace classification, or final answer/patch for downstream evaluation.3637## Scoring recipe3839```python40def compute_metrics(tp, fp, tn, fn):41 specificity = tn / (fp + tn)42 balanced_acc = (tn / (2 * (fp + tn))) + (tp / (2 * (tn + tp)))43 precision = tp / (tp + fp)44 recall = tp / (tp + fn)45 f05 = (1 + 0.5**2) * (precision * recall) / (0.5**2 * precision + recall)46 return specificity, balanced_acc, f0547```4849## Common pitfalls5051- Long reasoning traces are truncated to fit model context windows, which may alter correctness classification.52- Intermediate reasoning errors do not always lead to incorrect final answers, creating a tradeoff between specificity and recall.53- SWE-Bench unit tests are computationally expensive, preventing full benchmark evaluation during RL training loops.5455## Evidence (verbatim from paper)5657> For experiments assessing rubric quality as an artifact for trace classification, we consider the following metrics: Specificity, or $\frac{TN}{FP+TN}$ (the percentage of incorrect traces classified as incorrect by the LLM), balanced accuracy, or $\frac{TN}{2(FP+TN)}+\frac{TP}{2(TN+TP)}$ (the average of the specificity and recall), and F0.5, $(1+\beta^{2})\frac{PR}{\beta^{2}P+R}$ where $\beta\=0.5$, $P\=\frac{TP}{TP+FP}$, and $R$ is recall.5859## Citation6061```bibtex62@misc{sanders2026generating,63 title={Generating Data-Driven Reasoning Rubrics for Domain-Adaptive Reward Modeling},64 author={Sanders et al. (2026)},65 year={2026},66 note={arXiv:2602.06795}67}68```6970- arXiv: 2602.06795
qhjqhj00