Chronos Forecasting Eval

chronos-forecasting-eval

Chronos: Learning the Language of Time Series — Ansari et al. (2024) (arXiv:2403.07815, 2024)

What this evaluates

Evaluates time series forecasting models on in-domain and zero-shot benchmarks across diverse domains and frequencies. It probes a model's ability to generalize to unseen temporal patterns using both probabilistic and point forecast metrics.

Datasets

• Benchmark I — total 97272; splits: test (-1)
• Benchmark II — total 190674; splits: test (-1)

Metrics

• WQL (primary) — range: other
  • Weighted Quantile Loss computed over 9 uniformly-spaced quantile levels {0.1, ..., 0.9}. Measures compatibility between predictive distribution and ground truth.
• MASE — range: other
  • Mean Absolute Scaled Error: absolute forecast error scaled by the historical seasonal error of the time series. For probabilistic models, the median (0.5-quantile) forecast is used.

Input / output format

Input: Scaled and quantized time series history (context length 512) tokenized into a fixed vocabulary, passed through a T5 or GPT-2 architecture.

Output: Forecast tokens for the next H time steps, decoded back to continuous values. Probabilistic outputs are provided as quantiles at levels {0.1, ..., 0.9}.

Scoring recipe

relative_scores = []
for dataset in datasets:
    model_score = compute_metric(dataset, model_preds) # WQL or MASE
    baseline_score = compute_metric(dataset, seasonal_naive_preds)
    relative_scores.append(model_score / baseline_score)
aggregated_score = exp(mean(log(relative_scores)))
return aggregated_score

Common pitfalls

• Aggregating relative scores across datasets using an arithmetic mean instead of the geometric mean, which the authors note yields misleading conclusions.
• Comparing zero-shot models that were inadvertently trained on some Benchmark II datasets (pretraining contamination).
• Using raw WQL/MASE values directly for cross-dataset comparison without normalizing against the Seasonal Naive baseline.

Evidence (verbatim from paper)

Whenever possible, we evaluated models both in terms of their probabilistic and point forecast performance. We used the weighted quantile loss (WQL) to assess the quality of the probabilistic forecasts: the WQL is related to the continuous ranked probability score (CRPS, Gneiting & Raftery (2007)) and is commonly used to evaluate probabilistic forecasts (Gasthaus et al., 2019; Shchur et al., 2023). The WQL measures the compatibility between the predictive distribution and the ground-truth observation at a uniformly-spaced grid of quantile levels; we compute the WQL on 9 uniformly-spaced quantile levels {0.1, 0.2, ..., 0.9}. Quantile forecasters such as TFT were directly trained on these quantile levels. For methods requiring sampling, we estimated the quantiles using 20 sample forecast paths. We used the mean absolute scaled error (MASE, Hyndman & Koehler (2006)) to evaluate the point forecast performance. The MASE is defined as the absolute error of the forecast scaled by the historical seasonal error of the time series, and was selected due to its favorable properties over other point forecasting metrics (Hyndman & Koehler, 2006). We used the median forecast (0.5-quantile) for co

Citation

@misc{ansari2024chronos,
  title={Chronos: Learning the Language of Time Series},
  author={Ansari et al. (2024)},
  year={2024},
  note={arXiv:2403.07815}
}

• arXiv: 2403.07815