Medfuse Eval

medfuse-eval

MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images — Hayat et al. (2022) (arXiv:2207.07027, 2022)

What this evaluates

Evaluates a multi-modal fusion model's ability to predict patient phenotypes and in-hospital mortality using partially paired clinical time-series data and chest X-ray images. It probes robustness to missing modalities and temporal alignment in ICU settings.

Datasets

• MIMIC-IV / MIMIC-CXR — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/nyuad-cai/MedFuse

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the trade-off between true positive rate and false positive rate across all classification thresholds.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall curve. Measures the trade-off between precision and recall across all classification thresholds, particularly sensitive to class imbalance.

Input / output format

Input: Clinical time-series data vector x_ehr in R^(t x 76) (regularly sampled every 2 hours, standardized continuous features, one-hot encoded categorical features) paired with a chest X-ray image x_cxr.

Output: Binary or multi-label vector y_ehr (25 binary labels for phenotype classification, or 1 binary label for in-hospital mortality).

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score

def compute_metrics(y_true, y_pred, task):
    if task == 'phenotype':
        auroc = roc_auc_score(y_true, y_pred, average='macro', multi_label=True)
        auprc = average_precision_score(y_true, y_pred, average='macro')
    else:
        auroc = roc_auc_score(y_true, y_pred)
        auprc = average_precision_score(y_true, y_pred)
    return auroc, auprc

# Bootstrap 1000 iterations for 95% CI
scores = [compute_metrics(y_true, y_pred, task) for _ in range(1000)]

Common pitfalls

• Splits are performed at the patient level to prevent data leakage across ICU stays, not at the sample level.
• The 25 phenotype labels require mapping ICD-10 codes to ICD-9 and then to CCS categories, differing from the original Harutyunyan et al. (2019) benchmark.
• Models must handle partially paired data where chest X-rays are missing, often requiring learnable substitute vectors or masking strategies.

Evidence (verbatim from paper)

We evaluate this task using the Area Under the Receiver Operating Characteristic (AUROC) curve and the Area Under the Precision Recall curve (AUPRC). Using the patient identifier of the clinical time-series data, we randomly split the dataset into 70% for training, 10% for validation, and 20% for test set, as shown in Figure 1. We report final results on the test sets and compute 95% confidence intervals with 1000 iterations via the bootstrap method (Efron and Tibshirani, 1994).

Citation

@misc{hayat2022medfuse,
  title={MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images},
  author={Hayat et al. (2022)},
  year={2022},
  note={arXiv:2207.07027}
}

• arXiv: 2207.07027