Lane Level Traffic Prediction Eval

lane-level-traffic-prediction-eval

Unifying Lane-Level Traffic Prediction from a Graph Structural Perspective: Benchmark and Baseline — Shuhao Li et al. (arXiv:2403.14941, 2024)

What this evaluates

Evaluates models' ability to predict lane-level traffic speed and flow by modeling spatio-temporal dependencies on graph-structured lane networks. It tests performance across both regular and irregular lane configurations, emphasizing both predictive accuracy and training efficiency.

Datasets

• PeMS — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ShuhaoLii/TITS24LaneLevel-Traffic-Benchmark
• PeMSF — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ShuhaoLii/TITS24LaneLevel-Traffic-Benchmark
• HuaNan — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ShuhaoLii/TITS24LaneLevel-Traffic-Benchmark

Metrics

• MAE (primary) — range: [0, ∞)
  • MAE = (1/z) * Σ_{t=1}^z (1/N) * Σ_{j=1}^J (1/I) * Σ_{i=1}^I |y_t^{l_{i,j}} - ŷ_t^{l_{i,j}}|, where z is prediction horizon, N is number of time steps, J is lanes, I is segments.
• RMSE — range: [0, ∞)
  • RMSE = sqrt((1/z) * Σ_{t=1}^z (1/N) * Σ_{j=1}^J (1/I) * Σ_{i=1}^I (y_t^{l_{i,j}} - ŷ_t^{l_{i,j}})^2).
• MAPE — range: percent
  • MAPE = (1/z) * Σ_{t=1}^z (1/N) * Σ_{j=1}^J (1/I) * Σ_{i=1}^I |y_t^{l_{i,j}} - ŷ_t^{l_{i,j}}| / y_t^{l_{i,j}}.
• Cost — range: seconds
  • Training expenditure metric calculated as the time required for each iteration of model training multiplied by 100 (or 10^-2 per table), unit in seconds.

Input / output format

Input: Time-series speed and flow measurements from traffic sensors across multiple lanes, organized according to the physical lane topology (graph structure).

Output: Predicted speed and flow values for each lane segment at specified future time horizons (e.g., 3, 6, 12 steps).

Scoring recipe

def compute_metrics(y_true, y_pred, z, N, J, I):
    # y_true, y_pred shape: (z, N, J, I)
    mae = np.mean(np.abs(y_true - y_pred))
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
    return mae, rmse, mape

Common pitfalls

• Models must handle irregular lane counts (e.g., entrance lanes adding a 6th lane) without breaking graph topology assumptions.
• Training cost is evaluated as a first-class metric (time per iteration * 10^-2), not just accuracy; ignoring efficiency misrepresents practical utility.
• Missing data is imputed using adjacent time slot means, which may smooth out sudden traffic spikes and affect MAPE/RMSE.

Evidence (verbatim from paper)

In the field of traffic prediction, particularly at the lane level, a model’s accuracy is typically assessed using three key metrics: Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE). However, beyond accuracy, the practicality of a model also hinges on the duration of its training. This is especially crucial in lane-level traffic prediction scenarios where rapid response and real-time updates are essential. Therefore, focusing solely on predictive accuracy while overlooking training time could limit the model’s performance in practical applications. Acknowledging this, we have included training expenditure as an evaluation metric, aiming to ensure that models consider both training time and predictive accuracy. The training Cost metric is calculated as the time required for each iteration of model training multiplied by 100, with the unit being seconds.

Citation

@misc{li2024unifying,
  title={Unifying Lane-Level Traffic Prediction from a Graph Structural Perspective: Benchmark and Baseline},
  author={Shuhao Li et al.},
  year={2024},
  note={arXiv:2403.14941}
}

• arXiv: 2403.14941