Simbarca Traffic Forecasting Eval

simbarca-traffic-forecasting-eval

Multi-Source Urban Traffic Flow Forecasting with Drone and Loop Detector Data — Xiong et al. (2025) (arXiv:2501.03492, 2025)

What this evaluates

Evaluates the ability of deep learning models to forecast urban traffic speeds at both the individual road segment and regional levels. It probes spatio-temporal forecasting capabilities under varying congestion levels, testing how well models integrate multi-source sensor data (drone trajectories and loop detectors) to predict future traffic states.

Datasets

• SimBarca — total 101; splits: train (75), test (26)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: average of absolute differences between predicted and true speeds across all time steps and segments. Formula: MAE = (1/n) Σ |ŷ_i - y_i|.
• RMSE — range: other
  • Root Mean Square Error: square root of the average of squared differences between predicted and true speeds. Formula: RMSE = sqrt((1/n) Σ (ŷ_i - y_i)^2).
• MAPE* — range: percent
  • Modified Mean Absolute Percentage Error: average of absolute percentage errors, computed only where true speed > 1 m/s to avoid division by zero. Formula: MAPE* = (1/n) Σ |(ŷ_i - y_i)/y_i| for y_i > 1 m/s.

Input / output format

Input: Time-series sequences of road segment speeds (from drones, sampled every 5s) and point speeds (from loop detectors, aggregated to 3-min intervals) over a 30-minute historical window, combined with an undirected graph representation of the road network (1570 nodes, 2803 edges).

Output: Predicted future traffic speeds (in m/s) for specific road segments and aggregated spatial regions over 15-minute or 30-minute horizons (5 or 10 time steps).

Scoring recipe

def compute_metrics(pred, true):
    # pred, true: arrays of shape (num_segments, num_steps)
    # Metrics are calculated per segment/region, then averaged
    mae = np.mean(np.abs(pred - true))
    rmse = np.sqrt(np.mean((pred - true)**2))
    mask = true > 1.0
    mape_star = np.mean(np.abs((pred[mask] - true[mask]) / true[mask])) * 100
    return mae, rmse, mape_star

Common pitfalls

• MAPE becomes infinite when true speed is 0; the paper explicitly uses MAPE* by filtering out samples where speed ≤ 1 m/s.
• Loop detector data (point speed) systematically overestimates true segment speed due to sensor placement relative to stop lines, which can bias single-modality baselines if not accounted for.
• Prediction horizons are fixed at 15 and 30 minutes, corresponding to 5 and 10 discrete time steps in the dataset, not continuous time intervals.

Evidence (verbatim from paper)

Following the common practice in traffic forecasting literature[[8]], the prediction results are evaluated with three metrics: MAE, Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE). Their vector forms for a pair of prediction ($\hat{\mathbf{y}}$) and ground truth ($\mathbf{y}$) are defined as follows: [formula] where $n$ is an index for flattened predictions and labels. In the evaluation, the speed values take the unit of m/s, and the metrics are calculated for each road segment (or region) and then averaged over all segments (regions). Since a zero segment speed in the label will result in infinite MAPE, we only evaluate MAPE when the speed value is greater than 1 m/s, and we refer to this modified metric as MAPE*.

Citation

@misc{xiong2025simbarca,
  title={Multi-Source Urban Traffic Flow Forecasting with Drone and Loop Detector Data},
  author={Xiong et al. (2025)},
  year={2025},
  note={arXiv:2501.03492}
}

• arXiv: 2501.03492