Medprmbench Eval

medprmbench-eval

MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning — Wu et al. (2026) (arXiv:2604.17282, 2026)

What this evaluates

This benchmark evaluates the ability of process reward models and general critic models to detect factual, logical, and clinical errors at individual reasoning steps in medical question-answering. It probes step-level correctness verification and case-level chain validation, emphasizing the identification of clinically critical mistakes such as missing contraindications, flawed diagnostic logic, or premature conclusions.

Datasets

• MedPRMBench — total 13000; splits: train (-1), test (-1)

Metrics

• PRMScore (primary) — range: percent
  • PRMScore = 0.5 * F1_neg + 0.5 * F1, where F1 treats erroneous steps as the positive class and F1_neg treats correct steps as the positive class. It provides a unified, normalized score that balances error detection with correct-step preservation.

Input / output format

Input: A medical question Q paired with a step-by-step reasoning chain S={s1,...,sn}, where each reasoning step is separated by a delimiter token (e.g., 'ки' for open-source models or numbered formatting for API models).

Output: For open-source models: token probabilities for '+' and '-' at each delimiter position. For API/critic models: a sequence of '+' (correct) or '-' (erroneous) symbols, one per step.

Scoring recipe

step_preds = ['correct' if p_plus >= 0.5 else 'erroneous' for p_plus in model_probs]
case_pred = 'correct' if all(p == 'correct' for p in step_preds) else 'erroneous'
f1_pos = f1_score(gold_labels, step_preds, pos_label='erroneous')
f1_neg = f1_score(gold_labels, step_preds, pos_label='correct')
prmscore = 0.5 * f1_neg + 0.5 * f1_pos

Common pitfalls

• Case-level evaluation requires ALL steps in a chain to be predicted correctly for the chain to be labeled correct; a single step error makes the entire chain erroneous.
• Open-source models use a 0.5 probability threshold on token outputs, while proprietary/API models use a generative protocol outputting explicit '+' or '-' symbols, requiring different parsing and evaluation logic.
• PRMScore equally weights F1 for correct and erroneous steps, so models with strong positive or negative bias will score poorly even if they excel at detecting one class.

Evidence (verbatim from paper)

Following PRMBench (Song et al., 2025), we adopt PRMScore as the primary evaluation metric, defined as: PRM-Score = w1 * F1_neg + w2 * F1 where F1 treats erroneous steps as the positive class and F1_neg treats correct steps as the positive class. We set w1 = w2 = 0.5 to weight both classes equally, since a reliable PRM must excel at both detecting erroneous steps and preserving correct ones—over-weighting either side would mask critical failure modes.

Citation

@misc{wu2026medprmbench,
  title={MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning},
  author={Wu et al. (2026)},
  year={2026},
  note={arXiv:2604.17282}
}

• arXiv: 2604.17282