Clinicrealm Eval

clinicrealm-eval

ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks — Yinghao Zhu et al. (2024) (arXiv:2407.18525, 2024)

What this evaluates

Evaluates clinical prediction capabilities on unstructured notes and structured EHR data. It benchmarks zero-shot LLMs, finetuned BERTs, and conventional ML/DL models on mortality, readmission, and length-of-stay prediction tasks. The setup tests out-of-the-box prompting versus task-specific finetuning across diverse model families.

Datasets

• MIMIC-IV — total ?; splits: test (-1)
• TJH — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• AUPRC — range: [0, 1]
  • Area under the Precision-Recall Curve, summarizing precision-recall trade-offs, particularly useful for imbalanced datasets.
• MAE — range: other
  • Mean Absolute Error, the average of absolute differences between predicted and actual values for regression tasks like Length of Stay.
• MSE — range: other
  • Mean Squared Error, the average of squared differences between predicted and actual values.
• RMSE — range: other
  • Root Mean Squared Error, the square root of MSE, providing error magnitude in the same units as the target variable.

Input / output format

Input: Unstructured clinical notes (text) or structured Electronic Health Record (EHR) features (tabular/codified patient data).

Output: Binary classification probability or label (mortality/readmission) or continuous regression value (Length of Stay).

Scoring recipe

def compute_metrics(predictions, gold):
    auc = roc_auc_score(gold, predictions)
    aupr = average_precision_score(gold, predictions)
    mae = mean_absolute_error(gold, predictions)
    mse = mean_squared_error(gold, predictions)
    rmse = np.sqrt(mse)
    return {'AUROC': auc, 'AUPRC': aupr, 'MAE': mae, 'MSE': mse, 'RMSE': rmse}

Common pitfalls

• All metric values in the tables are multiplied by 100 for readability; divide by 100 to get standard [0,1] or raw scale values.
• Standard deviations are reported via bootstrapping over all test set samples 100 times, not standard cross-validation splits.
• Zero-shot prompting settings for LLMs are evaluated differently than finetuned settings; some smaller LLMs failed to parse prompts and were excluded.

Evidence (verbatim from paper)

We use a bootstrapping strategy on all test set samples 100 times to report the mean±std results. All metrics are multiplied by 100 for readability purposes. For mortality prediction, DeepSeek-R1 achieved the highest AUROC (97.64%), closely followed by o3-mini-high (97.58%) and DeepSeek-V3 (97.13%).

Citation

@misc{zhu2024clinicrealm,
  title={ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks},
  author={Yinghao Zhu et al. (2024)},
  year={2024},
  note={arXiv:2407.18525}
}

• arXiv: 2407.18525