Toto TS Forecasting Eval

toto-ts-forecasting-eval

This Time is Different: An Observability Perspective on Time Series Foundation Models — Ben Cohen et al. (2025) (arXiv:2505.14766, 2025)

What this evaluates

Evaluates zero-shot and fine-tuned time series forecasting capabilities on real-world observability telemetry and general-purpose benchmarks. Probes model robustness to high-dimensional, nonstationary, multivariate series with skewed distributions and varying temporal intervals.

Datasets

• Boom — total 350000000; splits: test (-1)
• Boomlet — total 23000000; splits: test (-1)
• GIFT-Eval — total 158000000; splits: test (-1)
• LSF — total 11000000; splits: train (-1), test (-1)

Metrics

• CRPS (primary) — range: [0, ∞)
  • Continuous Ranked Probability Score. Normalized by the Seasonal Naive forecast baseline and aggregated across tasks using a geometric mean.
• MASE — range: [0, ∞)
  • Mean Absolute Scaled Error. Normalized by the Seasonal Naive forecast baseline and aggregated across tasks using a shifted geometric mean.
• Rank — range: [1, N]
  • Mean rank across all forecasting tasks with respect to CRPS. Lower is better.
• MAE — range: [0, ∞)
  • Mean Absolute Error. Reported per dataset and averaged across LSF tasks.
• MSE — range: [0, ∞)
  • Mean Squared Error. Reported per dataset and averaged across LSF tasks.

Input / output format

Input: Historical multivariate time series data with varying intervals (10 sec to 1 day/yr) and series lengths (101 to 16,384 points). Input is processed via per-variate patch scaling.

Output: Forecasted values for a specified prediction length (48 to 900 points), output as point estimates or probabilistic distributions (Student-T mixture).

Scoring recipe

def aggregate_metrics(task_results):
    crps_vals = [r['crps'] for r in task_results.values()]
    mase_vals = [r['mase'] for r in task_results.values()]
    crps_agg = exp(mean(log(crps_vals)))
    mase_agg = exp(mean(log(mase_vals)))
    ranks = [rank_in_list(crps_vals, r['crps']) for r in task_results.values()]
    rank_agg = mean(ranks)
    return {'CRPS': crps_agg, 'MASE': mase_agg, 'Rank': rank_agg}

Common pitfalls

• Data leakage in competing models on GIFT-Eval due to partial training data overlap with the benchmark.
• Inference speed and lack of batch support prevent some models (e.g., TabPFN) from running on large benchmarks like Boom.
• Full-shot deep learning models are often impractical at scale for observability data, making zero-shot comparison more relevant for this domain.

Evidence (verbatim from paper)

MASE and CRPS are normalized by the Seasonal Naive forecast and aggregated across tasks using shifted geometric mean. Rank is the mean rank across tasks with respect to CRPS.

Citation

@misc{cohen2025observability,
  title={This Time is Different: An Observability Perspective on Time Series Foundation Models},
  author={Ben Cohen et al. (2025)},
  year={2025},
  note={arXiv:2505.14766}
}

• arXiv: 2505.14766