TS Forecasting Eval

ts-forecasting-eval

What Matters in Deep Learning for Time Series Forecasting? — Moretti et al. (2025) (arXiv:2512.22702, 2025)

What this evaluates

Evaluates the forecasting accuracy and computational efficiency of deep learning models on multivariate time series data. It probes how architectural choices, preprocessing steps, and spatial-temporal processing configurations impact performance across varying forecasting horizons.

Datasets

• Weather — total ?; splits: (unstated)
• Solar-Energy — total ?; splits: (unstated)
• ECL — total ?; splits: (unstated)
• Traffic — total ?; splits: (unstated)

Metrics

• MSE — range: other
  • Mean Squared Error: the average of the squared differences between predicted and actual values. Formula: (1/N) * Σ(y_true - y_pred)².
• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual values. Formula: (1/N) * Σ|y_true - y_pred|.

Input / output format

Input: Multivariate time series windows of configurable length, optionally augmented with exogenous covariates.

Output: Predicted values for the next horizon time steps for each variable in the series.

Scoring recipe

def compute_mae(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))

def compute_mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

Common pitfalls

• Performance gains are often attributed to novel architectural components rather than implementation choices like channel-independence or attention across channels.
• Computational efficiency metrics (batch time, GPU memory) are reported but frequently ignored when comparing model rankings.
• Forecasting horizon length drastically changes model rankings; results at H=96 do not generalize to H=720.

Evidence (verbatim from paper)

The results in [Tab. 8] extend[Tab. 1] for both MSE and MAE. We added in[Tab. 9], a comparison of all the possible configurations—local, global, and hybrid—for linear models. The results are consistent with the findings discussed in[Sec. 4.1].

Citation

@misc{moretti2025whatmatters,
  title={What Matters in Deep Learning for Time Series Forecasting?},
  author={Moretti et al. (2025)},
  year={2025},
  note={arXiv:2512.22702}
}

• arXiv: 2512.22702