Traffic Speed Forecasting Eval

traffic-speed-forecasting-eval

Traffic Flow Forecasting with Maintenance Downtime via Multi-Channel Attention-Based Spatio-Temporal Graph Convolutional Networks — Yuanjie Lu et al. (arXiv:2110.01535, 2021)

What this evaluates

This evaluation protocol assesses a model's ability to forecast future traffic speeds on road networks under varying conditions, including the impact of construction workzones. It probes spatio-temporal dependency modeling by measuring prediction accuracy across multiple forecast horizons (15, 30, and 60 minutes) on real-world highway sensor data.

Datasets

• Tyson's Corner — total ?; splits: train (-1), val (-1), test (-1)
• Los-loop — total ?; splits: train (-1), val (-1), test (-1)
• PEMS-BAY — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• RMSE (primary) — range: MPH
  • Root Mean Squared Error: the square root of the mean of the squared differences between predicted and true traffic speeds. Measures absolute error magnitude in MPH.
• MAE — range: MPH
  • Mean Absolute Error: the mean of the absolute differences between predicted and true traffic speeds. Measures average absolute error magnitude in MPH.
• MAPE — range: percent
  • Mean Absolute Percentage Error: the mean of the absolute percentage differences between predicted and true traffic speeds. Measures relative error as a percentage.

Input / output format

Input: Time-series traffic speed and workzone/maintenance features for graph nodes (road segments), along with the road network topology (adjacency matrix). Data is normalized using Min-Max scaling.

Output: Predicted traffic speed values (in MPH) for each road segment at specified future time steps (15, 30, or 60 minutes ahead).

Scoring recipe

def compute_metrics(y_true, y_pred):
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    mae = np.mean(np.abs(y_true - y_pred))
    mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
    return {'RMSE': rmse, 'MAE': mae, 'MAPE': mape}

Common pitfalls

• Reporting minimum RMSE instead of mean RMSE, which artificially inflates performance and contradicts the paper's explicit instruction to use mean RMSE.
• Failing to exclude accident-affected time steps from the Tyson's dataset, which introduces noise not accounted for in the protocol.
• Comparing models trained with workzone data against those without on datasets lacking workzone records without using the ablated variant (GCN-RWZ-).

Evidence (verbatim from paper)

Traffic speed prediction performance was quantified using three metrics: Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE). Note that since our prediction is traffic speed, the unit for RMSE and MAE, which measure the error between predicted and ground-truth/true speed in a network segment, is miles per hour (MPH). In contrast, MAPE considers not only the error between the predicted and true speed, but also the ratio of the error to the true value. The smaller MAPE is, the better the prediction performance of a model.

Citation

@misc{lu2021trafficflow,
  title={Traffic Flow Forecasting with Maintenance Downtime via Multi-Channel Attention-Based Spatio-Temporal Graph Convolutional Networks},
  author={Yuanjie Lu et al.},
  year={2021},
  note={arXiv:2110.01535}
}

• arXiv: 2110.01535