Hierarchical Time Series Forecasting Eval

hierarchical-time-series-forecasting-eval

Graph-based Time Series Clustering for End-to-End Hierarchical Forecasting — Cini et al. (2023) (arXiv:2305.19183, 2023)

What this evaluates

Evaluates the ability of spatiotemporal graph neural networks to perform multistep-ahead forecasting on correlated time series while simultaneously learning hierarchical cluster structures end-to-end. It probes the model's capacity to leverage relational inductive biases and self-supervised aggregation for improved prediction accuracy.

Datasets

• METR-LA — total ?; splits: train/val/test (-1)
• PEMS-BAY — total ?; splits: train/val/test (-1)
• AQI — total ?; splits: train/val/test (-1)
• CER-E — total ?; splits: train/val/test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: average of absolute differences between predicted and true values over the forecasting horizon and all nodes.
• MRE — range: other
  • Mean Relative Error: average of relative differences between predicted and true values. Often computed as MAE divided by the mean of true values or element-wise relative error averaged.

Input / output format

Input: Multivariate time series sequences of fixed length (input window) with associated graph adjacency matrices representing spatial correlations.

Output: Predicted multivariate time series values for a fixed forecasting horizon length.

Scoring recipe

def compute_mae(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))

def compute_mre(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred) / np.abs(y_true))

Common pitfalls

• Splits and graph topology extraction protocols are inherited from prior works ([24], [63], [22]) and not detailed in this paper.
• MRE definition is not explicitly formulaic in the text; readers should verify if it uses element-wise relative error or global mean normalization.
• Cluster quality is evaluated qualitatively via visualization (Fig. 3) rather than quantitative clustering metrics.

Evidence (verbatim from paper)

The performance metrics employed are the mean absolute error(MAE) and the mean relative error(MRE). The considered benchmarks comprise of 4 datasets in total and include 2 datasets from the traffic forecasting literature (METR-LA and PEMS-BAY), one dataset of air quality measurements (AQI) and a collection of energy consumption profiles (CER-E).

Citation

@misc{cini2023graph,
  title={Graph-based Time Series Clustering for End-to-End Hierarchical Forecasting},
  author={Cini et al. (2023)},
  year={2023},
  note={arXiv:2305.19183}
}

• arXiv: 2305.19183