Timer Time Series Eval

timer-time-series-eval

Timer: Generative Pre-trained Transformers Are Large Time Series Models — Liu et al. (2024) (arXiv:2402.02368, 2024)

What this evaluates

Evaluates a large decoder-only Transformer model on standard time series benchmarks for forecasting, imputation, and anomaly detection. It probes the model's few-shot generalization, scalability, and robustness in data-scarce scenarios compared to encoder-only baselines.

Datasets

• ETT — total ?; splits: test (-1)
• ECL — total ?; splits: test (-1)
• Traffic — total ?; splits: test (-1)
• Weather — total ?; splits: test (-1)
• PEMS — total ?; splits: test (-1)
• UCR Anomaly Archive — total ?; splits: test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between predicted and ground truth values, calculated as the average of squared differences.
• MAE — range: other
  • Mean Absolute Error between predicted and ground truth values, calculated as the average of absolute differences.

Input / output format

Input: Time series data tokenized into a unified single-series sequence (S3) format. Forecasting uses a fixed lookback window of 672 time points; imputation uses segments of length 24 with some fully masked; anomaly detection uses observed segments to predict the next segment.

Output: Autoregressive next-token predictions corresponding to the forecasted, imputed, or anomalous segment values.

Scoring recipe

def compute_metrics(pred, true):
    mse = np.mean((pred - true) ** 2)
    mae = np.mean(np.abs(pred - true))
    return mse, mae

Common pitfalls

• Strictly exclude downstream datasets from the pre-training corpus (UTSD) to prevent data leakage.
• Maintain the fixed lookback length of 672 and forecast length of 96 for forecasting tasks.
• Replicate few-shot scenarios using exact data scarcity ratios (1%, 5%, 20%, 100%) rather than full training sets.

Evidence (verbatim from paper)

We adopt the unified lookback length as $672$ and the forecast length as $96$. We pre-training Timer on UTSD-12G with the segment length $S=96$ and the number of tokens $N=15$, such that Timer can deal with time series with the context length up to $1440$. And the downstream forecasting task is naturally converted into the next token prediction as detailed in Appendix[B.2]. Table 2: In-domain and out-of-domain forecasting results by pre-training on the source domain and fine-tuning on the target dataset under different data scarcity. ... Metric | MSE | MAE

Citation

@misc{liu2024timer,
  title={Timer: Generative Pre-trained Transformers Are Large Time Series Models},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2402.02368}
}

• arXiv: 2402.02368