Dl Traff Eval

dl-traff-eval

DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction — Jiang et al. (2021) (arXiv:2108.09091, 2021)

What this evaluates

Evaluates the predictive accuracy and computational efficiency of deep learning models for urban traffic forecasting. It benchmarks grid-based, graph-based, and multivariate time-series architectures on standard traffic datasets to compare their ability to capture spatiotemporal dependencies.

Datasets

• BikeNYC-I — total ?; splits: train (-1), val (-1), test (-1)
• TaxiNYC — total ?; splits: train (-1), val (-1), test (-1)
• TaxiBJ — total ?; splits: train (-1), val (-1), test (-1)
• METR-LA — total ?; splits: train (-1), val (-1), test (-1)
• PeMS-BAY — total ?; splits: train (-1), val (-1), test (-1)
• PEMSD7M — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and actual values. Used as the default loss function during training.
• RMSE — range: other
  • Root Mean Square Error: the square root of the average of squared differences between predicted and actual values.
• MAPE — range: percent
  • Mean Absolute Percentage Error: the average of absolute percentage differences between predicted and actual values, expressed as a percentage.

Input / output format

Input: Historical traffic measurements (volume or speed) over a fixed observation window, structured as grid cells or graph nodes with temporal dimensions.

Output: Predicted traffic measurements for a specified prediction horizon (1 step for grid models; 1, 2, or 3 steps for graph models).

Scoring recipe

def compute_metrics(y_true, y_pred):
    mask = y_true != 0
    y_t, y_p = y_true[mask], y_pred[mask]
    mae = np.mean(np.abs(y_t - y_p))
    rmse = np.sqrt(np.mean((y_t - y_p) ** 2))
    mape = np.mean(np.abs((y_t - y_p) / y_t)) * 100
    return {'MAE': mae, 'RMSE': rmse, 'MAPE': mape}

Common pitfalls

• Zero values in the ground truth are explicitly ignored when computing MAE, RMSE, and MAPE.
• Grid-based and graph-based models use different fixed observation/prediction step settings (6 vs 12), so direct comparison requires matching these horizons.
• Training uses early stopping if validation error converges within 10 epochs, otherwise runs for a maximum of 200 epochs; the best validation model is saved.

Evidence (verbatim from paper)

Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) are used as metrics, where zero values will be ignored.

Citation

@misc{jiang2021dltraff,
  title={DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction},
  author={Jiang et al. (2021)},
  year={2021},
  note={arXiv:2108.09091}
}

• arXiv: 2108.09091