Contextformer Eval

contextformer-eval

Context Matters: Leveraging Contextual Features for Time Series Forecasting — Chattopadhyay et al. (2024) (arXiv:2410.12672, 2024)

What this evaluates

Evaluates whether integrating multimodal contextual metadata into pre-trained time series forecasting models improves prediction accuracy. It probes the model's ability to align external covariates with historical time series data to enhance forecast precision across multiple domains and horizons.

Datasets

• Synthetic ARMA(2,2) — total ?; splits: test (-1)
• PEMS-SF — total ?; splits: test (-1)
• ETT (ETTm2) — total ?; splits: test (-1)
• ECL — total ?; splits: test (-1)
• Beijing AQ — total ?; splits: test (-1)
• Store Sales — total ?; splits: test (-1)
• Monash (Bitcoin) — total ?; splits: test (-1)
• Bitcoin + News — total ?; splits: test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error: average of the squared differences between predicted and actual values. Standard regression loss for time series forecasting.
• MAE — range: other
  • Mean Absolute Error: average of the absolute differences between predicted and actual values.

Input / output format

Input: Time series history of length L=96 concatenated with contextual metadata (continuous, categorical, time-varying, or 1536-dim textual embeddings).

Output: Forecasted time series values for horizon T (T ∈ {48, 96} or T=24 for Bitcoin).

Scoring recipe

import numpy as np
def compute_metrics(preds, gold):
    mae = np.mean(np.abs(preds - gold))
    mse = np.mean((preds - gold) ** 2)
    return {'MAE': mae, 'MSE': mse}

Common pitfalls

• Results in Table 2 are reported on a normalized scale for cross-dataset comparison, while Table 4 uses original units; mixing these scales causes incorrect performance interpretation.
• The method relies on plug-and-play fine-tuning rather than training from scratch; evaluating a fully trained context-aware model yields worse results than the fine-tuned approach.
• Context-agnostic baselines (PatchTST, iTransformer) must be compared against their own ContextFormer-enhanced versions, not just against context-aware models like TiDE or TimeXer.

Evidence (verbatim from paper)

Forecast accuracy was evaluated using Mean Squared Error (MSE) and Mean Absolute Error (MAE) as performance metrics. The ContextFormer-enhanced models were benchmarked against their respective base architectures and state-of-the-art context-aware forecasters like TimeXer and TiDE.

Citation

@misc{chattopadhyay2024contextmatters,
  title={Context Matters: Leveraging Contextual Features for Time Series Forecasting},
  author={Chattopadhyay et al. (2024)},
  year={2024},
  note={arXiv:2410.12672}
}

• arXiv: 2410.12672