Vitalbench Eval

vitalbench-eval

VitalBench: A Rigorous Multi-Center Benchmark for Long-Term Vital Sign Prediction in Intraoperative Care — Cai et al. (2025) (arXiv:2511.13757, 2025)

What this evaluates

Evaluates long-term multivariate time-series forecasting of intraoperative vital signs under three clinically realistic conditions: complete data, variable missingness, and cross-center generalization. It probes a model's ability to handle heterogeneous clinical data, adapt to missing sensor inputs, and generalize across different hospital centers.

Datasets

• VitalDB — total 962; splits: train (-1), val (-1), test (-1); repo https://github.com/XiudingCai/VitalBench
• MOVER-SIS — total 3221; splits: train (-1), val (-1), test (-1); repo https://github.com/XiudingCai/VitalBench

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: average of absolute differences between predicted and actual values. Lower is better.
• RMSE — range: other
  • Root Mean Squared Error: square root of the average of squared differences between predicted and actual values. Lower is better.
• R2 — range: [0, 1]
  • R-squared (Coefficient of Determination): proportion of variance in the target variable explained by the model. Higher is better.
• CC — range: [0, 1]
  • Correlation Coefficient (Pearson): measures linear correlation between predicted and actual values. Higher is better.

Input / output format

Input: Multivariate time-series sequences of dynamic physiological variables and static covariates over a 30-minute retrospective window (1 Hz sampling). Input dimensions may vary across samples due to randomly dropped variable channels in Tracks 2 and 3.

Output: Predicted time-series values for target vital signs over specified prediction horizons (1, 3, 10, or 30 minutes).

Scoring recipe

def compute_metrics(pred, gold):
    mae = np.mean(np.abs(pred - gold))
    rmse = np.sqrt(np.mean((pred - gold) ** 2))
    ss_res = np.sum((gold - pred) ** 2)
    ss_tot = np.sum((gold - np.mean(gold)) ** 2)
    r2 = 1 - (ss_res / ss_tot)
    cc = np.corrcoef(pred, gold)[0, 1]
    return {'MAE': mae, 'RMSE': rmse, 'R2': r2, 'CC': cc}

Common pitfalls

• Applying fixed-dimension models (e.g., LightTS, MICN) to Tracks 2 and 3, which require handling variable input dimensions due to missing sensor channels.
• Imputing missing values before training or evaluation, which introduces spurious data and biases predictions compared to the benchmark's recommended masked loss approach.
• Using random data splitting instead of chronological splitting, which violates the temporal validity requirement for intraoperative time-series forecasting.

Evidence (verbatim from paper)

For all experiments, we used standard evaluation metrics, including Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), R-squared (R2), and Correlation Coefficient (CC), to ensure a comprehensive and consistent assessment of model performance. Lower values for MAE and RMSE indicate better accuracy, while higher values for R2 and CC suggest improved model fit and correlation.

Citation

@misc{cai2025vitalbench,
  title={VitalBench: A Rigorous Multi-Center Benchmark for Long-Term Vital Sign Prediction in Intraoperative Care},
  author={Cai et al. (2025)},
  year={2025},
  note={arXiv:2511.13757}
}

• arXiv: 2511.13757