Ppg Health Benchmark Eval

ppg-health-benchmark-eval

Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals — Kataria et al. (2025) (arXiv:2510.14254, 2025)

What this evaluates

Evaluates the transferability and emergent capabilities of generalist and specialist time-series foundation models on diverse physiological tasks using PPG and cross-modal signals. Probes classification (e.g., arrhythmia, mental load, activity recognition) and regression (e.g., vital signs, blood chemistry, blood pressure) performance across clinical and ambulatory settings.

Datasets

• Stanford AF — total 41114; splits: train (-1), val (-1), test (-1)
• Simband — total 857; splits: train (-1), val (-1), test (-1)
• Real World PPG — total 2074; splits: train (1374), test (700)
• MIMIC-III — total ?; splits: train (-1), val (-1), test (-1)
• Sleep-EDF — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted class labels out of total samples. Calculated as correct predictions divided by total test samples.
• MAE — range: [0, inf)
  • Mean Absolute Error between predicted and ground-truth continuous values. Standard metric for regression tasks in this benchmark.

Input / output format

Input: Fixed-length PPG signal segments (resampled to 40 Hz, min-max normalized to [0,1]) or cross-modal physiological signals (EEG, CBFV) depending on the downstream task. Segment duration varies by dataset (e.g., 5s, 10s, 25s, 30s).

Output: Classification tasks: discrete class labels (e.g., AF/non-AF, stress/non-stress, sleep/wake). Regression tasks: continuous physiological values (e.g., SpO2%, electrolyte concentration, heart rate, blood pressure, ICP).

Scoring recipe

def compute_metrics(predictions, gold_labels, task_type):
    if task_type == 'classification':
        correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
        return correct / len(gold_labels)
    elif task_type == 'regression':
        errors = [abs(p - g) for p, g in zip(predictions, gold_labels)]
        return sum(errors) / len(errors)

Common pitfalls

• Data leakage from overlapping time windows: PPG segments must strictly precede lab results to preserve temporal causality.
• Participant overlap: Inter-patient splits are required; mixing participants across train/test invalidates generalization claims.
• Signal normalization & resampling: All segments must be min-max normalized and resampled to 40 Hz before evaluation to match model training conditions.

Evidence (verbatim from paper)

The tasks are broadly categorized into two types: classification and regression. ... To ensure consistency across samples, we apply min-max normalization to each segment, scaling the data within the range of 0 to 1. ... we employ an inter-patient evaluation scheme in which the training and testing data are derived from different groups of participants to assess the model’s generalizability. ... We report top-1 accuracy on the official test set.

Citation

@misc{kataria2025generalist,
  title={Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals},
  author={Kataria et al. (2025)},
  year={2025},
  note={arXiv:2510.14254}
}

• arXiv: 2510.14254