Cloudops Forecasting Eval

cloudops-forecasting-eval

Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain — Woo et al. (2023) (arXiv:2310.05063, 2023)

What this evaluates

Evaluates time series forecasting models, particularly pre-trained Transformers, on cloud operations data. It probes zero-shot generalization, architectural efficiency, and scaling behavior against classical and deep learning baselines.

Datasets

• azure2017 — total ?; splits: test (-1)
• borg2011 — total ?; splits: test (-1)
• ali2018 — total ?; splits: test (-1)

Metrics

• sMAPE (primary) — range: percent
  • Symmetric Mean Absolute Percentage Error. Computed as the average of |y - ŷ| / ((|y| + |ŷ|)/2) over all time steps. Lower is better.
• CRPS — range: other
  • Continuous Ranked Probability Score, measuring the accuracy of probabilistic forecasts against observed values. Lower is better.

Input / output format

Input: Time series context window of length L=480 containing target values and covariates.

Output: Probabilistic forecast distribution parameters (e.g., Student-T parameters) for the prediction horizon.

Scoring recipe

def compute_smape(y_true, y_pred):
    diff = np.abs(y_true - y_pred)
    scale = (np.abs(y_true) + np.abs(y_pred)) / 2
    return np.mean(diff / scale) * 100

def compute_crps(y_true, dist_params):
    # CRPS for Student-T distribution
    return crps_student_t(y_true, dist_params)

Common pitfalls

• Fine-tuning pre-trained models often yields no benefit over zero-shot inference due to sufficient pre-training diversity.
• Date/time features are not critical for forecasting in this domain; RoPE positional encodings perform better.
• The naive forecast serves as a surprisingly strong baseline for high-frequency CloudOps data.

Evidence (verbatim from paper)

We observe that our zero-shot approach constitutes a very strong baseline, obtaining a 27/24% reduction in sMAPE/CRPS from the next best performing method on the largest dataset, azure2017, generally outperforming all other methods.

Citation

@misc{woo2023pushing,
  title={Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain},
  author={Woo et al. (2023)},
  year={2023},
  note={arXiv:2310.05063}
}

• arXiv: 2310.05063