qhjqhj00 quito-bench-eval
QuitoBench: A High-Quality Open Time Series Forecasting Benchmark — Xue et al. (2026) (arXiv:2603.26017, 2026)
What this evaluates
Evaluates time series forecasting models on a regime-balanced benchmark stratified by trend, seasonality, and forecastability. It probes a model's ability to handle varying context lengths, forecast horizons, and multivariate dependencies while mitigating domain bias and information leakage.
Datasets
- QuitoBench — total ?; splits: train (-1), val (-1), test (1290); repo https://github.com/alipay/quito
Metrics
MAE(primary) — range: other- Mean Absolute Error: the average of the absolute differences between predicted and actual values over all time steps and series.
Input / output format
Input: A contiguous window of a time series of length L (96, 576, or 1024) used to predict the next H steps (48, 288, or 512). Input is either multivariate (all 5 channels jointly) or univariate (each channel independently).
Output: A sequence of H predicted values corresponding to the forecast horizon.
Scoring recipe
def compute_mae(predictions, targets):
return np.mean(np.abs(predictions - targets))
def compute_mean_rank(all_series_maes):
# Rank models 1-10 per series/config, then average ranks
ranks = rankdata(all_series_maes, method='min')
return np.mean(ranks)
Common pitfalls
- Using non-overlapping windows (stride H) instead of dense rolling windows (unit stride), which reduces evaluation instances by orders of magnitude and inflates variance.
- Averaging raw MAE across series with different scales without converting to ranks, which biases results toward high-magnitude series.
- Treating multivariate and univariate modes identically; MV predicts all channels jointly while UV processes them independently, leading to different performance rankings.
Evidence (verbatim from paper)
Metrics. We report MAE as the primary metric and convert per-series MAE to rank scores (1–10) to aggregate fairly across heterogeneous scales; the mean rank is reported over all series and configurations (See [section E.3] for details).
Citation
@misc{xue2026quito,
title={QuitoBench: A High-Quality Open Time Series Forecasting Benchmark},
author={Xue et al. (2026)},
year={2026},
note={arXiv:2603.26017}
}
- arXiv: 2603.26017