qhjqhj00 tempusbench-univariate-eval
TempusBench: An Evaluation Framework for Time-Series Forecasting — Goktas et al. (2026) (arXiv:2604.11529, 2026)
What this evaluates
Evaluates time-series foundation models, statistical methods, and machine learning algorithms on univariate forecasting tasks. It probes their ability to handle diverse statistical properties like stationarity, seasonality, sparsity, and noise across real-world and synthetic datasets.
Datasets
- TempusBench Univariate Benchmark — total ?; splits: test (-1); repo https://github.com/Smlcrm/TempusBench
Metrics
MASE(primary) — range: [0, ∞)- Mean Absolute Scaled Error; a value below 1.0 indicates the model outperforms a naive baseline forecast.
Input / output format
Input: Univariate time series sequences (single variable) with historical observations.
Output: Point forecasts for the target horizon.
Scoring recipe
def compute_mase(y_true, y_pred, y_train):
naive_errors = np.abs(y_train[1:] - y_train[:-1])
scale = np.mean(naive_errors)
forecast_errors = np.abs(y_true - y_pred)
return np.mean(forecast_errors) / scale
Common pitfalls
- Evaluating models without standardized hyperparameter tuning across statistical, ML, and DL types creates unfair performance comparisons.
- Ignoring core statistical properties like stationarity and seasonality when selecting or interpreting benchmark tasks leads to misleading generalization claims.
- Using a naive baseline that does not match the series' seasonal period distorts MASE interpretation.
Evidence (verbatim from paper)
The MASE (Mean Absolute Scaled Error) results reveal distinct performance patterns for single time series forecasting. TimesFM emerges as the most consistent top performer, achieving MASE < 1.0 (better than naive baseline) on 8 out of 26 tasks, including standout performances on Electricity (0.79971) and Soil Nature (0.27718).
Citation
@misc{goktas2026tempusbench,
title={TempusBench: An Evaluation Framework for Time-Series Forecasting},
author={Goktas et al. (2026)},
year={2026},
note={arXiv:2604.11529}
}
- arXiv: 2604.11529