Gift Eval Eval

gift-eval-eval

GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation — Aksu et al. (2024) (arXiv:2410.10393, 2024)

What this evaluates

Evaluates zero-shot time series forecasting models across diverse domains, frequencies, and prediction horizons. It probes a model's ability to generalize to unseen multivariate and univariate series, identifying strengths and weaknesses in short-term versus long-term forecasting.

Datasets

• GIFT-Eval — total 144000; splits: test (-1); repo https://github.com/SalesforceAIResearch/gift-eval

Metrics

• MAPE (primary) — range: percent
  • Median of the Mean Absolute Percentage Error across all series. Computed as median(mean(|(y_true - y_pred) / y_true|) * 100). Normalized against the Seasonal Naive baseline.
• CRPS — range: other
  • Continuous Ranked Probability Score evaluating the accuracy of probabilistic forecasts against observed values. Normalized against the Seasonal Naive baseline.

Input / output format

Input: Historical univariate or multivariate time series data across 7 domains, 10 frequencies, and varying prediction lengths.

Output: Point forecasts (single values) and/or probabilistic forecasts (full predictive distributions or samples).

Scoring recipe

def compute_metrics(y_true, y_pred, y_pred_dist, baseline_pred, baseline_dist):
    mape = median(mean(abs((y_true - y_pred) / y_true)) * 100)
    crps = continuous_ranked_probability_score(y_true, y_pred_dist)
    baseline_mape = median(mean(abs((y_true - baseline_pred) / y_true)) * 100)
    baseline_crps = continuous_ranked_probability_score(y_true, baseline_dist)
    norm_mape = mape / baseline_mape
    norm_crps = crps / baseline_crps
    return norm_mape, norm_crps

Common pitfalls

• Pre-training datasets of foundation models (TimesFM, Chronos, Moirai) exhibit partial data leakage into GIFT-Eval, requiring careful isolation or retraining.
• Decoder-only foundation models use recursive multi-step forecasting, causing severe error accumulation for medium/long-term predictions.
• Metrics must be normalized against the Seasonal Naive baseline to standardize comparisons across different benchmarks.

Evidence (verbatim from paper)

Performance is assessed using two metrics: the median Mean Absolute Percentage Error (MAPE) for point forecasts and the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery, 2007) for probabilistic forecasts (definition of both metrics are in Appendix C). To standardize comparison across benchmarks, both metrics are normalized against the Seasonal Naive baseline.

Citation

@misc{aksu2024gift_eval,
  title={GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation},
  author={Aksu et al. (2024)},
  year={2024},
  note={arXiv:2410.10393}
}

• arXiv: 2410.10393