Cura Mimic Iv Eval

cura-mimic-iv-eval

CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction — Wang et al. (arXiv:2604.14651, 2026)

What this evaluates

Probes a clinical language model's ability to predict binary adverse outcomes from free-text EHR notes while simultaneously calibrating its prediction uncertainty. It evaluates both discriminative accuracy and probabilistic calibration across multiple clinical risk stratification tasks.

Datasets

• MIMIC-IV — total ?; splits: 5-fold cross-validation (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• AUPRC — range: [0, 1]
  • Area under the Precision-Recall Curve, summarizing the trade-off between precision and recall across thresholds, particularly sensitive to class imbalance.
• Brier score — range: [0, 1]
  • Mean squared difference between predicted probabilities and actual binary outcomes: $\frac{1}{N}\sum_{i=1}^N (p_i - y_i)^2$. Lower is better.
• NLL — range: other
  • Negative Log-Likelihood: $-\frac{1}{N}\sum_{i=1}^N [y_i \log(p_i) + (1-y_i) \log(1-p_i)]$. Measures probabilistic calibration and sharpness.
• AURC — range: [0, 1]
  • Area Under the Risk-Coverage curve, evaluating how well uncertainty estimates correlate with prediction errors across varying coverage levels.

Input / output format

Input: Single free-text clinical note from electronic health records.

Output: Binary label $y \in {0, 1}$ indicating a pre-defined adverse outcome within a fixed horizon, along with a predicted probability/uncertainty score.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score

def compute_metrics(y_true, y_prob):
    auroc = roc_auc_score(y_true, y_prob)
    auprc = average_precision_score(y_true, y_prob)
    brier = np.mean((y_prob - y_true) ** 2)
    nll = -np.mean(y_true * np.log(y_prob + 1e-8) + (1 - y_true) * np.log(1 - y_prob + 1e-8))
    sorted_idx = np.argsort(y_prob)
    coverage = np.arange(1, len(y_true) + 1) / len(y_true)
    errors = 1 - y_true[sorted_idx]
    aurc = np.trapz(errors, coverage)
    return {'AUROC': auroc, 'AUPRC': auprc, 'Brier': brier, 'NLL': nll, 'AURC': aurc}

Common pitfalls

• Results are averaged over 5 cross-validation folds, not a single held-out test split; failing to report fold-wise variance obscures result stability.
• Calibration metrics (Brier, NLL, AURC) require well-calibrated predicted probabilities, not hard class predictions or logits.
• AURC evaluates uncertainty alignment across varying coverage levels, not at a fixed decision threshold like 0.5.

Evidence (verbatim from paper)

We evaluate model performance using two categories of metrics: (1) Discrimination: Area Under the ROC Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC); and (2) Calibration & Uncertainty: Brier score, Negative Log-Likelihood (NLL), and Area Under the Risk-Coverage curve (AURC). All reported results represent the mean and standard deviation across five cross-validation folds.

Citation

@misc{wang2026cura,
  title={CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction},
  author={Wang et al.},
  year={2026},
  note={arXiv:2604.14651}
}

• arXiv: 2604.14651