Epidemiological Benchmark Eval

epidemiological-benchmark-eval

Synthetic Datasets for Machine Learning on Spatio-Temporal Graphs using PDEs — Arndt et al. (2025) (arXiv:2502.04140, 2025)

What this evaluates

Evaluates spatio-temporal graph models on forecasting, stability, and denoising tasks using synthetic epidemiological data generated from PDEs. Probes the model's ability to predict future states, resist noise/dropout, and recover clean signals from corrupted graph time-series.

Datasets

• Epidemiological Synthetic Dataset — total ?; splits: train (-1), val (-1); repo https://github.com/Jostarndt/Synthetic_Datasets_for_Temporal_Graphs

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error, calculated as the square root of the average of squared differences between predicted and true values across all samples and time-steps.

Input / output format

Input: Historical spatio-temporal graph node features over time, optionally corrupted with additive Gaussian noise or dropout for stability/denoising tasks.

Output: Predicted node features for subsequent time-steps.

Scoring recipe

def compute_rmse(predictions, targets):
    # predictions, targets: arrays of shape (num_samples, num_time_steps)
    squared_diff = (predictions - targets) ** 2
    mean_squared_error = np.mean(squared_diff)
    return np.sqrt(mean_squared_error)

Common pitfalls

• RMSE is reported per time-step across the validation set rather than as a single aggregate scalar, requiring careful per-step plotting or aggregation.
• Stability and denoising experiments require applying specific noise distributions (Gaussian or Dropout) to the training data, which must exactly match the evaluation setup.
• Models are trained on synthetic data and evaluated on synthetic validation data for stability/denoising, not real-world data, which limits direct real-world performance claims.

Evidence (verbatim from paper)

A visual comparison of RMSEs for each time-step in the forecasting experiment for different models can be seen in Fig[10]... Figure 11: Plots of the RMSE of the presented models during the stability experiments for each predicted time-step on the validation dataset. The models were trained on the synthetic dataset, and evaluated on a synthetic dataset with: a) additive Gaussian noise, b) dropout noise.

Citation

@misc{arndt2025synthetic,
  title={Synthetic Datasets for Machine Learning on Spatio-Temporal Graphs using PDEs},
  author={Arndt et al. (2025)},
  year={2025},
  note={arXiv:2502.04140}
}

• arXiv: 2502.04140