Time Moe Forecasting Eval

time-moe-forecasting-eval

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts — Xiaoming Shi et al. (2024) (arXiv:2409.16040, 2024)

What this evaluates

Evaluates long-term time series forecasting capabilities of foundation models in both zero-shot (unseen datasets) and in-distribution (fine-tuned) settings across multiple prediction horizons.

Datasets

• ETTh1 — total ?; splits: train (-1), test (-1)
• ETTh2 — total ?; splits: train (-1), test (-1)
• ETTm1 — total ?; splits: train (-1), test (-1)
• ETTm2 — total ?; splits: train (-1), test (-1)
• Weather — total ?; splits: train (-1), test (-1)
• Global Temp — total ?; splits: train (-1), test (-1)

Metrics

• MSE (primary) — range: [0, inf)
  • Mean Squared Error: average of squared differences between predicted and actual values. Lower is better.
• MAE — range: [0, inf)
  • Mean Absolute Error: average of absolute differences between predicted and actual values. Lower is better.

Input / output format

Input: Time series sequences of fixed lengths (512, 1024, 2048, or 3072) corresponding to prediction horizons (96, 192, 336, or 720).

Output: Forecasted time series values for the specified prediction horizon.

Scoring recipe

def compute_metrics(pred, true):
    mse = np.mean((true - pred) ** 2)
    mae = np.mean(np.abs(true - pred))
    return {'MSE': mse, 'MAE': mae}

Common pitfalls

• TimesFM is excluded from the Weather dataset evaluation due to pretraining data overlap.
• In-distribution (full-shot) evaluation strictly uses only one fine-tuning epoch.
• Input and output lengths are rigidly paired per horizon (e.g., 512 input for 96-step forecast).
• Reported averages in tables are computed across all six datasets and four horizons.

Evidence (verbatim from paper)

The evaluation metrics adopt mean square error (MSE) and mean absolute error (MAE).

Citation

@misc{shi2024timemoe,
  title={Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts},
  author={Xiaoming Shi et al. (2024)},
  year={2024},
  note={arXiv:2409.16040}
}

• arXiv: 2409.16040