# Tsfm Scaling Eval

> Evaluates how time series foundation models scale in forecasting accuracy and uncertainty calibration as model size, compute, and training data size increase. It probes both in-distribution generalization and out-of-distribution transfer capabilities across multiple standard time series forecasting benchmarks. Use when the user wants to benchmark on Monash subset, LSF subset, or asks about evaluating this task. Reports NLL.

- Skill: `qhjqhj00/tsfm-scaling-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/tsfm-scaling-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/tsfm-scaling-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/tsfm-scaling-eval

---


# tsfm-scaling-eval

> Towards Neural Scaling Laws for Time Series Foundation Models — Yao et al. (2024) (arXiv:2410.12360, 2024)

## What this evaluates

Evaluates how time series foundation models scale in forecasting accuracy and uncertainty calibration as model size, compute, and training data size increase. It probes both in-distribution generalization and out-of-distribution transfer capabilities across multiple standard time series forecasting benchmarks.

## Datasets

- **Monash subset** — total ?; splits: test (-1)
- **LSF subset** — total ?; splits: test (-1)

## Metrics

- `NLL` **(primary)** — range: other
  - Negative log-likelihood of the true values under the model's predictive distribution. Lower is better.
- `MAPE` — range: percent
  - Mean Absolute Percentage Error: average of |(actual - predicted) / actual| over the forecasting horizon.
- `SMAPE` — range: percent
  - Symmetric Mean Absolute Percentage Error: 2*|actual - predicted| / (|actual| + |predicted|), averaged over the horizon.
- `MASE` — range: other
  - Mean Absolute Scaled Error: ratio of MAE to the in-sample naive forecast MAE.
- `CRPS` — range: other
  - Continuous Ranked Probability Score: integrated squared difference between the predictive CDF and the step function at the observed value.

## Input / output format

**Input**: Historical time series sequences (context window) provided to the foundation model for zero-shot or fine-tuned forecasting.

**Output**: Point forecasts or full predictive distributions (probability density functions) for the target forecasting horizon.

## Scoring recipe

```python
def compute_metrics(y_true, y_pred, y_pred_dist=None):
    mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
    smape = 2 * np.mean(np.abs(y_true - y_pred) / (np.abs(y_true) + np.abs(y_pred))) * 100
    mase = np.mean(np.abs(y_true - y_pred)) / np.mean(np.abs(y_true[1:] - y_true[:-1]))
    nll = -np.mean(y_pred_dist.log_prob(y_true)) if y_pred_dist else None
    crps = np.mean((y_pred_dist.cdf(y_true) - 0.5)**2) if y_pred_dist else None
    return {'MAPE': mape, 'SMAPE': smape, 'MASE': mase, 'NLL': nll, 'CRPS': crps}
```

## Common pitfalls

- OOD performance degrades predictably but scaling gains remain proportional across distributions, so absolute OOD scores should not be compared directly to ID scores.
- Different metrics exhibit distinct power-law exponents; scaling laws are metric-dependent and cannot be assumed to transfer across MAPE, NLL, etc.
- Performance improvements are not always smooth; 'emergent behaviors' cause abrupt, non-continuous jumps at specific model sizes (e.g., 10M parameters).

## Evidence (verbatim from paper)

> We investigate the scaling behaviors of five common performance metrics: NLL, MAPE, SMAPE, MASE, and CRPS, (as shown in Figures 15 - 19). All metrics exhibit a decreasing trend following an approximate power-law; however, each metric demonstrates distinct scaling characteristics, reflected in their varying power-law exponents.

## Citation

```bibtex
@misc{yao2024towards,
  title={Towards Neural Scaling Laws for Time Series Foundation Models},
  author={Yao et al. (2024)},
  year={2024},
  note={arXiv:2410.12360}
}
```

- arXiv: 2410.12360