Chaos Forecasting Eval

chaos-forecasting-eval

Chaos as an interpretable benchmark for forecasting and data-driven modelling — Gilpin (2021) (arXiv:2110.05266, 2021)

What this evaluates

Evaluates the ability of time series forecasting models to predict trajectories of low-dimensional chaotic dynamical systems. It probes how well models capture underlying deterministic chaos, smoothness, and multi-scale temporal dependencies without explicit trend or seasonality signals.

Datasets

• Chaotic Dynamical Systems Benchmark — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/williamgilpin/dysts

Metrics

• sMAPE (primary) — range: percent
  • Symmetric Mean Absolute Percentage Error. Computed as the average of the absolute difference between forecast and actual values divided by their sum, scaled by 200 to express as a percentage. The paper notes it is reported as the headline metric for model comparison.
• MSE — range: other
  • Mean Squared Error. The average of the squared differences between predicted and actual values. Included as a secondary metric alongside MASE, MAE, MARRE, |CV|, 1-r², MAPE, and sMAPE.

Input / output format

Input: Univariate or multivariate time series trajectories sampled from chaotic dynamical systems at specified granularities (coarse or fine). Models receive historical segments to generate forecasts for a specified prediction horizon.

Output: Predicted time series values for the target prediction horizon, aligned with the ground truth trajectory. Models are evaluated on unseen test trajectories emanating from different initial conditions than the training data.

Scoring recipe

def compute_smape(actual, predicted):
    import numpy as np
    actual = np.asarray(actual)
    predicted = np.asarray(predicted)
    numerator = np.abs(actual - predicted)
    denominator = np.abs(actual) + np.abs(predicted)
    denominator = np.where(denominator == 0, 1, denominator)
    error = 200 * np.mean(numerator / denominator)
    return error

Common pitfalls

• Using chronological train/test splits instead of differential initial conditions; the benchmark explicitly requires training on one trajectory and testing on a separate trajectory starting from a different initial condition.
• Assuming traditional statistical models (e.g., Prophet, ARIMA) will dominate; the smooth, continuous nature of chaotic attractors favors deep learning architectures like Transformers and NBEATS.
• Ignoring granularity and noise variations; model rankings remain stable across two orders of magnitude in sampling granularity and varying noise, but absolute error scales significantly.

Evidence (verbatim from paper)

For all forecasting models and dynamical systems we compute eight error metrics: the mean squared error (MSE), mean absolute scaled error (MASE), mean absolute error (MAE), mean absolute ranged relative error (MARRE), the magnitude of the coefficient of variation $(|CV|)$ , one minus the coefficient of determination $(1 - r^2)$ , and the symmetric and regular mean absolute percent errors (MAPE and sMAPE). We find that all of these potential metrics are positively correlated across our dataset, and that they can be grouped into families of strongly-related metrics (Figure 2B). We also observe that the relative ranking of different forecasting models is independent of the choice of metric. Hereafter, we report sMAPE errors when comparing models, but we include all other metrics within the benchmark.

Citation

@misc{gilpin2021chaos,
  title={Chaos as an interpretable benchmark for forecasting and data-driven modelling},
  author={Gilpin (2021)},
  year={2021},
  note={arXiv:2110.05266}
}

• arXiv: 2110.05266