Chaotic Time Series Forecasting Eval

chaotic-time-series-forecasting-eval

A projected nonlinear state-space model for forecasting time series signals — Donner et al. (2023) (arXiv:2311.13247, 2023)

What this evaluates

Evaluates the ability of time series forecasting models to predict future values of noisy, chaotic dynamical systems. It probes how well models capture underlying nonlinear dynamics and handle varying levels of observation noise and system complexity.

Datasets

• Gilpin chaotic systems benchmark — total ?; splits: train (1000), test (200)

Metrics

• SMAPE (primary) — range: percent
  • Symmetric Mean Absolute Percentage Error: 200 * mean(|z_t - z_hat_t| / (|z_t| + |z_hat_t|)), averaged over the forecasting period.

Input / output format

Input: Noisy univariate time series observations, often constructed via delayed coordinates of dimension D_x.

Output: Forecasted time series values for the subsequent 200 time steps.

Scoring recipe

def compute_smape(z_true, z_pred):
    numerator = np.abs(z_true - z_pred)
    denominator = np.abs(z_true) + np.abs(z_pred)
    denominator = np.where(denominator == 0, 1, denominator)
    return 200 * np.mean(numerator / denominator)

Common pitfalls

• Hyperparameters for baseline models (LSTM, Transformer, NBEATS) were optimized directly on the test dataset using SMAPE, which may inflate reported performance.
• The evaluation uses a fixed temporal split (first 1000 points for training, next 200 for testing) rather than a standard train/val/test partition, which can lead to high variance across systems with different lengths.

Evidence (verbatim from paper)

To noisy observations of the 127 chaotic dynamics we fitted fourteen different time series prediction methods (including the PNL-SS method) whose hyperparameters are optimized by test datasets using the symmetric mean absolute percent error (SMAPE) as an error measure. ... Here, the forecast error was measured by the symmetric mean absolute percentage error (SMAPE) defined as $200\langle{\left|z_{t}-\hat{z}{t}\right|/(\left|z{t}\right|+\left|\hat{z}{t}\right|)}\rangle$, where $z{t}$ is the test data and $\hat{z}_{t}$ is the forecasting values and $\langle\cdot\rangle$ denotes time average within a forecasting period (i.e., $t=1001,\ldots,1200$).

Citation

@misc{donner2023projected,
  title={A projected nonlinear state-space model for forecasting time series signals},
  author={Donner et al. (2023)},
  year={2023},
  note={arXiv:2311.13247}
}

• arXiv: 2311.13247