X Pcr Eval

x-pcr-eval

X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis — Gui Wang et al. (arXiv:2604.20350, 2026)

What this evaluates

Evaluates multi-modal large language models on progressive clinical reasoning in ophthalmic diagnosis. It tests the model's ability to perform a six-stage diagnostic chain (from image quality assessment to clinical decision-making) while integrating cross-modality imaging data and calibrating its uncertainty.

Datasets

• X-PCR — total 177868; splits: test (177868); repo https://github.com/CVI-SZU/X-PCR

Metrics

• Stage-Wise Accuracy (SWA) (primary) — range: percent
  • Percentage of correct answers at each of the six reasoning stages (IQA, AL, LC, DD, SG, CD).
• Chain Completion Rate (CCR) — range: percent
  • Percentage of instances where the model correctly answers all six stages in the reasoning chain.
• Expertise-Stratified Accuracy (ESA) — range: percent
  • Accuracy computed separately for three clinical difficulty tiers: Resident (R), Attending (A), and Specialist (S).
• Uncertainty-Aware Score (UAS) — range: other
  • Weighted score aggregating confidence-accuracy categories (CC, CU, IU, IC) with difficulty and clinical impact weights to reward calibrated confidence and penalize overconfident errors.
• Expected Calibration Error (ECE) — range: other
  • Bin-wise discrepancy between the model's self-reported confidence and its empirical accuracy across confidence bins.

Input / output format

Input: Multi-modal ophthalmic images (up to 6 modalities: EP, CFP, FFA, ICGA, OCT, RetCam) paired with clinical vignettes and stage-specific multiple-choice questions.

Output: Multiple-choice answer selection, self-reported confidence score [0,1], and step-by-step reasoning for each of the six clinical stages.

Scoring recipe

def compute_metrics(predictions, gold, confidences, difficulties):
    swa = {s: sum(p==g for p,g in zip(preds[s], gold[s]))/len(gold[s]) for s in stages}
    ccr = sum(all(p==g for p,g in zip(preds[s], gold[s])) for _ in instances)/len(instances)
    esa = {t: sum(p==g for p,g,d in zip(preds,gold,diff) if d==t)/count(t) for t in tiers}
    uas = sum(weight(cat(p,g,c))*impact(d) for p,g,c,d in zip(preds,gold,conf,diff))
    ece = sum(bin_size*abs(bin_acc-bin_conf) for bin in bins)
    return swa, ccr, esa, uas, ece

Common pitfalls

• Evaluating stages in isolation ignores the benchmark's core dependency-aware design, masking error propagation across the diagnostic chain.
• Reporting only standard accuracy overlooks the uncertainty calibration component; models frequently exhibit Incorrect Confident (IC) behavior, which heavily penalizes UAS.
• Failing to stratify results by difficulty tier (R/A/S) and clinical impact weights violates the expertise-aware assessment protocol.

Evidence (verbatim from paper)

We define the following evaluation metrics: 1) Stage-Wise Accuracy (SWA) per reasoning stage; 2) Chain Completion Rate (CCR) for fully correct six-stage chains; 3) Expertise-Stratified Accuracy (ESA) within each difficulty tier. ... Model responses include self-reported confidence (normalized to $[0,1]$), categorized as Correct Confident (CC), Correct Uncertain (CU), Incorrect Uncertain (IU), or Incorrect Confident (IC). Each category receives a base score, aggregated with difficulty–impact weights into a weighted Uncertainty-Aware Score (UAS), rewarding calibrated confidence and penalizing overconfident errors. We further report Expected Calibration Error (ECE)...

Citation

@misc{wang2026xpcr,
  title={X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis},
  author={Gui Wang et al.},
  year={2026},
  note={arXiv:2604.20350}
}

• arXiv: 2604.20350