Transport Forecasting Eval

transport-forecasting-eval

On the importance of stationarity, strong baselines and benchmarks in transport prediction problems — Rodrigues et al. (2022) (arXiv:2203.02954, 2022)

What this evaluates

Evaluates spatio-temporal forecasting models on traffic speed, volume, and bike flow prediction tasks. It specifically probes whether simple baselines that account for weekly stationarity (historical average plus linear regression on residuals) can match or outperform complex deep learning architectures across diverse transport datasets.

Datasets

• PeMSD7(M) — total ?; splits: train/val/test (-1)
• Urban1 — total ?; splits: train/val/test (-1)
• NYC Citi Bike — total ?; splits: train/val/test (-1)
• PeMSD4 — total ?; splits: train/val/test (-1)
• SZ-taxi — total ?; splits: train/val/test (-1)
• METR-LA — total ?; splits: train/val/test (-1)
• PEMS-BAY — total ?; splits: train/val/test (-1)
• NYC Bike in- and out-flows — total ?; splits: train/val/test (-1)
• Seattle traffic speeds — total ?; splits: train/val/test (-1)

Metrics

• RMSE (primary) — range: other
  • Root Mean Squared Error: sqrt(mean((y_true - y_pred)^2)). Lower values indicate better predictive accuracy.
• MAE — range: other
  • Mean Absolute Error: mean(|y_true - y_pred|). Lower values indicate better predictive accuracy.
• MAPE — range: percent
  • Mean Absolute Percentage Error: mean(|(y_true - y_pred) / y_true|) * 100. Lower values indicate better predictive accuracy.

Input / output format

Input: Spatio-temporal time series of traffic measurements (speed, volume, or bike flow counts) across multiple locations, typically using a fixed lookback window (e.g., 12 time steps).

Output: Point forecasts for future time steps (e.g., 15, 30, 45, 60 minutes or 1-3 hours ahead), with final scores averaged over multiple forecasting horizons.

Scoring recipe

def evaluate(preds, targets):
    mae = np.mean(np.abs(preds - targets))
    mape = np.mean(np.abs((preds - targets) / targets)) * 100
    rmse = np.sqrt(np.mean((preds - targets) ** 2))
    return {'MAE': mae, 'MAPE': mape, 'RMSE': rmse}

Common pitfalls

• Many baseline results are copied directly from original papers (marked with *) rather than re-run, risking unfair comparisons due to differing experimental setups or data preprocessing.
• Forecasting horizons vary significantly across datasets (from 15 minutes to 3 hours), complicating direct metric aggregation or cross-dataset comparison.
• The HA+LR baseline computes residuals on the training set historical average, which could inadvertently leak temporal information if the train/test split is not strictly chronological.

Evidence (verbatim from paper)

As it can be observed, a naive linear regression estimated on the previous $h=12$ residuals from the historical average (HA+LR) is able to obtain a lower RMSE for the 30 and 45min-ahead forecasting horizons, while being very competitive for the 15min forecasting horizon.

Citation

@misc{rodrigues2022stationarity,
  title={On the importance of stationarity, strong baselines and benchmarks in transport prediction problems},
  author={Rodrigues et al. (2022)},
  year={2022},
  note={arXiv:2203.02954}
}

• arXiv: 2203.02954