Time Series Benchmark Eval

time-series-benchmark-eval

It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks — Qiao et al. (2026) (arXiv:2602.12147, 2026)

What this evaluates

Evaluates zero-shot forecasting performance of time series foundation models across diverse datasets and horizons. Probes model capability to capture structural temporal patterns (trend, seasonality, stationarity, complexity) and generalizes to unseen data without leakage.

Datasets

• TIME Benchmark — total 50; splits: test (-1)

Metrics

• MASE (primary) — range: [0, inf)
  • Mean Absolute Scaled Error; measures point forecast accuracy by scaling the mean absolute error against the mean absolute error of a seasonal naive baseline.
• CRPS (primary) — range: [0, inf)
  • Continuous Ranked Probability Score; measures probabilistic forecast accuracy by integrating the squared difference between the predicted cumulative distribution function and the actual outcome.

Input / output format

Input: Historical time series window (univariate or multivariate) with a specified forecasting horizon.

Output: Point forecasts (single value per horizon step) or probabilistic forecasts (distribution samples or quantiles).

Scoring recipe

# 1. Compute raw metric per evaluation unit (task/variate)
raw_metric = compute_metric(predictions, ground_truth)
baseline_metric = compute_metric(seasonal_naive_predictions, ground_truth)

# 2. Normalize against baseline
norm_metric = raw_metric / baseline_metric

# 3. Aggregate across all units using geometric mean
final_score = geometric_mean(norm_metric_across_units)

# Note: For distribution-based models, sample 100 quantiles prior to metric computation.

Common pitfalls

• Metrics are normalized against a Seasonal Naive baseline; raw scores are not directly comparable across datasets.
• Geometric mean is used for aggregation, not arithmetic mean, to handle multiplicative relationships and outliers.
• Distribution-based models require sampling 100 quantiles prior to metric computation to match quantile-based models.

Evidence (verbatim from paper)

For metrics, we employ MASE and CRPS for point and probabilistic evaluation, respectively. ... we adopt the protocol in (Aksu et al., [2024]) and utilize a consistent relative evaluation framework where model metrics are normalized against a Seasonal Naive (S-Naive) baseline.

Citation

@misc{qiao2026time,
  title={It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks},
  author={Qiao et al. (2026)},
  year={2026},
  note={arXiv:2602.12147}
}

• arXiv: 2602.12147