Card Long Term Forecasting Eval

card-long-term-forecasting-eval

CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting — Wang Xue et al. (2023) (arXiv:2305.12095, 2023)

What this evaluates

Evaluates multivariate time series forecasting models on capturing temporal and cross-channel dependencies across multiple real-world benchmarks. It probes the model's ability to predict future values over varying horizons using fixed historical lookback windows.

Datasets

• ETTm1 — total ?; splits: test (-1)
• ETTm2 — total ?; splits: test (-1)
• ETTh1 — total ?; splits: test (-1)
• ETTh2 — total ?; splits: test (-1)
• Weather — total ?; splits: test (-1)
• Electricity — total ?; splits: test (-1)
• Traffic — total ?; splits: test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error: the average of the squared differences between predicted and actual values. Lower is better.
• MAE — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual values. Lower is better.

Input / output format

Input: Multivariate time series sequences with a fixed lookback length of 96 time steps.

Output: Predicted values for multiple forecasting horizons (96, 192, 336, 720 time steps).

Scoring recipe

def compute_metrics(y_true, y_pred):
    mse = np.mean((y_true - y_pred) ** 2)
    mae = np.mean(np.abs(y_true - y_pred))
    return mse, mae
# Final reported score = average of compute_metrics() over 10 independent random seeds and 4 prediction horizons (96, 192, 336, 720).

Common pitfalls

• Failing to apply reversible instance normalization (RevIN) to handle data heterogeneity, which breaks fair comparison with baselines.
• Using inconsistent lookback lengths across experiments, as the protocol strictly fixes the lookback at 96 for standard comparison.
• Reporting single-run results instead of averaging over 10 independent repeats as mandated by the evaluation protocol.

Evidence (verbatim from paper)

Each setting is repeated 10 times and average MSE/MAE results are reported. The results are summarized in Table 1. Regarding the average performance across four different output horizons, CARD gains the best performance in 6 out of 7 and 7 out of 7 in MSE and MAE, respectively.

Citation

@misc{wang2023card,
  title={CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting},
  author={Wang Xue et al. (2023)},
  year={2023},
  note={arXiv:2305.12095}
}

• arXiv: 2305.12095