Traffic Flow Forecasting Eval

traffic-flow-forecasting-eval

Graph Neural Network for spatiotemporal data: methods and applications — Yun Li et al. (2023) (arXiv:2306.00012, 2023)

What this evaluates

Evaluates the ability of spatiotemporal GNN models to forecast future traffic flow (speed or occupancy) based on historical sensor data and road network topology.

Datasets

• METR-LA — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/liyaguang/DCRNN
• PEMS-BAY — total ?; splits: train (-1), val (-1), test (-1); repo https://pems.dot.ca.gov/
• PeMS04 — total ?; splits: train (-1), val (-1), test (-1); repo https://pems.dot.ca.gov/

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: average of absolute differences between predicted and observed values.
• RMSE — range: other
  • Root Mean Square Error: square root of the average of squared differences between predicted and observed values, normalized by standard deviation in the paper's formula.
• MAPE — range: percent
  • Mean Absolute Percentage Error: average of absolute percentage differences between predicted and observed values.

Input / output format

Input: Historical traffic sensor readings (speed/occupancy) over a sequence of time steps, optionally with a pre-defined or learned adjacency matrix representing the road network.

Output: Predicted traffic flow values (speed or occupancy) for a specified future horizon (15, 30, or 60 minutes).

Scoring recipe

def compute_metrics(y_true, y_pred, sigma=None):
    n = len(y_true)
    mae = sum(abs(y - x) for x, y in zip(y_true, y_pred)) / n
    if sigma is not None:
        rmse = math.sqrt(sum(((y - x) / s) ** 2 for x, y, s in zip(y_true, y_pred, sigma)) / n)
    else:
        rmse = math.sqrt(sum((y - x) ** 2 for x, y in zip(y_true, y_pred)) / n)
    mape = sum(abs(y - x) / x for x, y in zip(y_true, y_pred)) / n
    return mae, rmse, mape

Common pitfalls

• The paper reports RMSE with a non-standard normalization by sigma_i (standard deviation), which differs from the typical RMSE formula.
• Datasets are split by time (e.g., 80% train, 10% val, 10% test) but exact split ratios are not explicitly stated in this section.
• Models are evaluated at multiple forecasting horizons (15, 30, 60 min), and results are not always aggregated into a single score.

Evidence (verbatim from paper)

These models are evaluated using metrics mean absolute error (MAE), Root-mean-square error(RMSE) and mean absolute percentage error (MAPE) where $x_{i}$ is the observed value and $y_{i}$ is the predicted value.

Citation

@misc{li2023gnnspatiotemporal,
  title={Graph Neural Network for spatiotemporal data: methods and applications},
  author={Yun Li et al. (2023)},
  year={2023},
  note={arXiv:2306.00012}
}

• arXiv: 2306.00012