Signalmc Med Eval

signalmc-med-eval

SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG — Gustafsson et al. (2026) (arXiv:2603.09940, 2026)

What this evaluates

This benchmark evaluates biosignal foundation models on synchronized, long-duration single-lead ECG and PPG recordings from emergency department visits. It probes the models' ability to extract clinically meaningful representations for tasks such as age and sex prediction, emergency disposition, laboratory value regression, and ICD-10 diagnosis classification.

Datasets

• SignalMC-MED — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/fregu856/SignalMC-MED

Metrics

• AUROC (primary) — range: other
  • Area under the receiver operating characteristic curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• Pearson correlation — range: other
  • Linear correlation coefficient between predicted and true continuous values. Ranges from -1 to 1, where 1 indicates perfect positive linear correlation.

Input / output format

Input: 10-minute single-lead ECG and/or PPG time-series signals, processed as non-overlapping 10-second segments to extract visit-level representations via mean aggregation.

Output: Predicted continuous values for regression tasks or class probabilities for classification tasks across 20 clinical benchmark tasks.

Scoring recipe

def score(predictions, gold, task_type):
    if task_type == 'regression':
        return pearsonr(predictions, gold)
    elif task_type == 'classification':
        return roc_auc_score(gold, predictions)
    return None

Common pitfalls

• Models are evaluated strictly as frozen feature extractors with linear probing, not fine-tuned end-to-end.
• Performance is aggregated across four training data percentages (10%, 25%, 50%, 100%) and averaged over five resampling repetitions, so single-run scores are not directly comparable.
• Multimodal fusion uses late feature-level averaging of independently extracted ECG and PPG representations, not joint multimodal training.

Evidence (verbatim from paper)

Performance is reported using Pearson correlation for regression and area under the receiver operating characteristic curve (AUROC) for classification. To assess robustness and data efficiency, downstream models are trained on 10%, 25%, 50%, and 100% of the available train visits with five repeated resamplings.

Citation

@misc{gustafsson2026signalmcmed,
  title={SignalMC-MED: A Multimodal Benchmark for Evaluating Biosignal Foundation Models on Single-Lead ECG and PPG},
  author={Gustafsson et al. (2026)},
  year={2026},
  note={arXiv:2603.09940}
}

• arXiv: 2603.09940