Tsfm Scaling Eval

tsfm-scaling-eval

Towards Neural Scaling Laws for Time Series Foundation Models — Yao et al. (2024) (arXiv:2410.12360, 2024)

What this evaluates

Evaluates how time series foundation models scale in forecasting accuracy and uncertainty calibration as model size, compute, and training data size increase. It probes both in-distribution generalization and out-of-distribution transfer capabilities across multiple standard time series forecasting benchmarks.

Datasets

• Monash subset — total ?; splits: test (-1)
• LSF subset — total ?; splits: test (-1)

Metrics

• NLL (primary) — range: other
  • Negative log-likelihood of the true values under the model's predictive distribution. Lower is better.
• MAPE — range: percent
  • Mean Absolute Percentage Error: average of |(actual - predicted) / actual| over the forecasting horizon.
• SMAPE — range: percent
  • Symmetric Mean Absolute Percentage Error: 2*|actual - predicted| / (|actual| + |predicted|), averaged over the horizon.
• MASE — range: other
  • Mean Absolute Scaled Error: ratio of MAE to the in-sample naive forecast MAE.
• CRPS — range: other
  • Continuous Ranked Probability Score: integrated squared difference between the predictive CDF and the step function at the observed value.

Input / output format

Input: Historical time series sequences (context window) provided to the foundation model for zero-shot or fine-tuned forecasting.

Output: Point forecasts or full predictive distributions (probability density functions) for the target forecasting horizon.

Scoring recipe

def compute_metrics(y_true, y_pred, y_pred_dist=None):
    mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
    smape = 2 * np.mean(np.abs(y_true - y_pred) / (np.abs(y_true) + np.abs(y_pred))) * 100
    mase = np.mean(np.abs(y_true - y_pred)) / np.mean(np.abs(y_true[1:] - y_true[:-1]))
    nll = -np.mean(y_pred_dist.log_prob(y_true)) if y_pred_dist else None
    crps = np.mean((y_pred_dist.cdf(y_true) - 0.5)**2) if y_pred_dist else None
    return {'MAPE': mape, 'SMAPE': smape, 'MASE': mase, 'NLL': nll, 'CRPS': crps}

Common pitfalls

• OOD performance degrades predictably but scaling gains remain proportional across distributions, so absolute OOD scores should not be compared directly to ID scores.
• Different metrics exhibit distinct power-law exponents; scaling laws are metric-dependent and cannot be assumed to transfer across MAPE, NLL, etc.
• Performance improvements are not always smooth; 'emergent behaviors' cause abrupt, non-continuous jumps at specific model sizes (e.g., 10M parameters).

Evidence (verbatim from paper)

We investigate the scaling behaviors of five common performance metrics: NLL, MAPE, SMAPE, MASE, and CRPS, (as shown in Figures 15 - 19). All metrics exhibit a decreasing trend following an approximate power-law; however, each metric demonstrates distinct scaling characteristics, reflected in their varying power-law exponents.

Citation

@misc{yao2024towards,
  title={Towards Neural Scaling Laws for Time Series Foundation Models},
  author={Yao et al. (2024)},
  year={2024},
  note={arXiv:2410.12360}
}

• arXiv: 2410.12360