Time Series Forecasting Eval

time-series-forecasting-eval

Adapting LLMs to Time Series Forecasting via Temporal Heterogeneity Modeling and Semantic Alignment — Yanru Sun et al. (arXiv:2508.07195, 2025)

What this evaluates

Evaluates the capability of LLM-based models to forecast multivariate time series across multiple prediction horizons and real-world datasets. It probes pattern-aware temporal modeling and semantic alignment by measuring prediction accuracy under a channel-independent, rolling forecasting setup.

Datasets

• ETTh1 — total ?; splits: train/val/test (-1)
• ETTh2 — total ?; splits: train/val/test (-1)
• ETTm1 — total ?; splits: train/val/test (-1)
• ETTm2 — total ?; splits: train/val/test (-1)
• Weather — total ?; splits: train/val/test (-1)
• ECL — total ?; splits: train/val/test (-1)
• Traffic — total ?; splits: train/val/test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error: the average of the squared differences between predicted and actual values. Formula: MSE = (1/n) Σ(y_true - y_pred)².
• MAE — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual values. Formula: MAE = (1/n) Σ|y_true - y_pred|.

Input / output format

Input: Multivariate time series data, channel-independently decomposed into univariate sequences and segmented into non-overlapping patches of length S.

Output: Forecasted values for a specified prediction horizon (96, 192, 336, or 720 steps) generated autoregressively.

Scoring recipe

def compute_metrics(y_true, y_pred):
    mse = np.mean((y_true - y_pred) ** 2)
    mae = np.mean(np.abs(y_true - y_pred))
    return mse, mae

Common pitfalls

• Rolling forecasting is used: a single model trained on a 96-step horizon is evaluated on longer horizons (192, 336, 720) without retraining.
• Channel-independent strategy is applied: each variable is modeled separately before aggregation, which may mask cross-variable dependencies.
• IMP score is reported as the average MSE and MAE reduction over baselines across seven datasets, not a direct model metric.

Evidence (verbatim from paper)

Multivariate forecasting (672-pred-{96, 192, 336, 720}) results under the one-for-all setting. Following (Liu et al. [2024c]), a single model is trained on a 96-step prediction horizon and evaluated on all horizons using rolling forecasting. The best results are in bold, and the second-best are underlined. Averaged results are reported here and full results are provided in Appendix. IMP denotes the average MSE and MAE reduction of TALON over each baseline across seven datasets.

Citation

@misc{sun2025talon,
  title={Adapting LLMs to Time Series Forecasting via Temporal Heterogeneity Modeling and Semantic Alignment},
  author={Yanru Sun et al.},
  year={2025},
  note={arXiv:2508.07195}
}

• arXiv: 2508.07195