qhjqhj00 mimic-cdm-eval
Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks — Wang et al. (2026) (arXiv:2603.12875, 2026)
What this evaluates
Evaluates large language models on clinical decision-making tasks by predicting diagnoses from structured patient evidence. It probes whether models possess latent domain-specific reasoning capabilities that are masked by unfamiliarity with benchmark input formats and task definitions.
Datasets
- MIMIC-CDM — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly predicted diagnoses out of the total number of clinical cases evaluated.
Input / output format
Input: A single prompt containing all available clinical information for a case (patient history, physical examination, lab results, and imaging reports). Sections exceeding 1,000 tokens are pre-summarized using GPT5-nano.
Output: A direct prediction of the underlying pathology or diagnosis.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Direct evaluation without task alignment systematically underestimates model capabilities due to unfamiliarity with domain-specific formats.
- Long clinical sections (e.g., imaging reports) exceed context limits and require external summarization (GPT5-nano) before model input.
- Task alignment via TTRA can drastically alter relative model rankings, making base models appear comparable to fine-tuned or human baselines.
Evidence (verbatim from paper)
We consider the MIMIC-CDM full-information(MIMIC-CDM-FI) setup of the benchmark, where we evaluate the models by providing all available clinical information in a single prompt. Figure 8 shows the clinical decision-making accuracy of several base models before and after alignment with TTRA, alongside a human doctor baseline reported in[[13]]. Without alignment, the Gemma base models[[32]] perform poorly. For instance, Gemma-3-1B reaches only 21.9% accuracy, which is worse than random guessing. After applying TTRA, all Gemma models improve substantially.
Citation
@misc{wang2026testtime,
title={Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks},
author={Wang et al. (2026)},
year={2026},
note={arXiv:2603.12875}
}
- arXiv: 2603.12875