Ehr Clinical Outcome Prediction Eval

ehr-clinical-outcome-prediction-eval

Cross-Representation Benchmarking in Time-Series Electronic Health Records for Clinical Outcome Prediction — Chen et al. (2025) (arXiv:2510.09159, 2025)

What this evaluates

This benchmark evaluates clinical outcome prediction models across three distinct EHR data representations (multivariate time-series, event streams, and textual event streams). It probes how well different architectures handle sparse, irregular longitudinal patient data and varying feature missingness rates in both acute ICU and long-term care settings.

Datasets

• MIMIC-IV — total ?; splits: train (-1), val (-1), test (-1)
• EHRSHOT — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 score, AUROC, AUPRC (primary) — range: [0, 1]
  • For binary tasks, F1, AUROC, and AUPRC are computed using a 0.5 probability threshold. For multi-label phenotyping, macro-averaged versions (macro-AUROC, macro-AUPRC, macro-F1) are reported.

Input / output format

Input: Structured EHR data formatted as either a multivariate time-series matrix, an OMOP-coded event stream, or a textual event stream converted for LLM ingestion.

Output: Binary or multi-label clinical outcome predictions (e.g., ICU mortality, ICU phenotyping, 30-day readmission, 1-year pancreatic cancer risk).

Scoring recipe

def compute_metrics(y_true, y_pred_proba, task_type='binary'):
    if task_type == 'binary':
        y_pred = (y_pred_proba >= 0.5).astype(int)
        f1 = f1_score(y_true, y_pred)
        auroc = roc_auc_score(y_true, y_pred_proba)
        auprc = average_precision_score(y_true, y_pred_proba)
        return {'F1': f1, 'AUROC': auroc, 'AUPRC': auprc}
    else:
        y_pred = (y_pred_proba >= 0.5).astype(int)
        return {
            'macro-AUROC': roc_auc_score(y_true, y_pred_proba, average='macro'),
            'macro-AUPRC': average_precision_score(y_true, y_pred_proba, average='macro'),
            'macro-F1': f1_score(y_true, y_pred, average='macro')
        }

Common pitfalls

• Assuming feature pruning always improves performance; missingness-based pruning helps ICU models but degrades longitudinal care prediction where sparse features carry important long-term information.
• Assuming pretrained foundation models always outperform simple baselines; count-based models (XGBoost on event counts) match or exceed pretrained models when sufficient training data is available.
• Ignoring representation-specific feature selection; time-series models require task-specific feature selection to handle sparsity, whereas event-stream models leverage all available features without optimization.

Evidence (verbatim from paper)

For binary tasks, we report AUROC, AUPRC, and F1 using a 0.5 threshold. For multi-label phenotyping, we report macro-AUROC, macro-AUPRC, and macro-F1.

Citation

@misc{chen2025crossrepresentationbenchmarking,
  title={Cross-Representation Benchmarking in Time-Series Electronic Health Records for Clinical Outcome Prediction},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2510.09159}
}

• arXiv: 2510.09159