Physiome Ode Eval

physiome-ode-eval

Physiome-ODE: A Benchmark for Irregularly Sampled Multivariate Time Series Forecasting Based on Biological ODEs — Klötergens et al. (2025) (arXiv:2502.07489, 2025)

What this evaluates

Evaluates the ability of models to forecast irregularly sampled multivariate time series generated from biological ordinary differential equations. It probes how well different architectures handle sparse, non-uniform temporal observations and varying levels of dynamical complexity.

Datasets

• Physiome-ODE — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Square Error: the average of the squared differences between predicted and true values across all time steps and channels. Formula: MSE = (1/N) Σ(y_true - y_pred)².

Input / output format

Input: Irregularly sampled multivariate time series containing the first 50% of observed time points and channels.

Output: Predicted values for the remaining 50% of time points across all target channels.

Scoring recipe

def compute_mse(predictions, targets):
    squared_diff = (predictions - targets) ** 2
    mse = np.mean(squared_diff)
    return mse

Common pitfalls

• A naive constant baseline (predicting a fixed value) can outperform complex neural ODE models on certain datasets, misleading accuracy comparisons.
• MSE has a theoretical lower bound determined by the observation noise variance (0.05), so values near 0.0025 indicate near-perfect performance rather than model failure.
• Dataset difficulty varies significantly; raw MSE scores alone do not capture dynamical complexity, necessitating auxiliary metrics like Joint Gradient Deviation (JGD) for meaningful interpretation.

Evidence (verbatim from paper)

In our experiments models have to predict the last 50% of the time series after observing the first 50%. We use 5-fold cross validation, and for each fold we split the data into training, validation and test set with a ratio of 70:20:10. Additionally, we resample the sparse observation mask to transform each instance into an IMTS. The evaluation metric is the mean square error (MSE). For simplicity, we randomly sample 10 hyperparameter configurations and fitted each model on a single fold per dataset, selecting the configuration with the lowest validation MSE. The winning configuration per model is then trained and evaluated on all 5 folds (see Appendix F).

Citation

@misc{klotergens2025physiomeode,
  title={Physiome-ODE: A Benchmark for Irregularly Sampled Multivariate Time Series Forecasting Based on Biological ODEs},
  author={Klötergens et al. (2025)},
  year={2025},
  note={arXiv:2502.07489}
}

• arXiv: 2502.07489