# Simbarca Traffic Forecasting Eval

> Evaluates the ability of deep learning models to forecast urban traffic speeds at both the individual road segment and regional levels. It probes spatio-temporal forecasting capabilities under varying congestion levels, testing how well models integrate multi-source sensor data (drone trajectories and loop detectors) to predict future traffic states. Use when the user wants to benchmark on SimBarca, or asks about evaluating this task. Reports MAE.

- Skill: `qhjqhj00/simbarca-traffic-forecasting-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/simbarca-traffic-forecasting-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/simbarca-traffic-forecasting-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/simbarca-traffic-forecasting-eval

---


# simbarca-traffic-forecasting-eval

> Multi-Source Urban Traffic Flow Forecasting with Drone and Loop Detector Data — Xiong et al. (2025) (arXiv:2501.03492, 2025)

## What this evaluates

Evaluates the ability of deep learning models to forecast urban traffic speeds at both the individual road segment and regional levels. It probes spatio-temporal forecasting capabilities under varying congestion levels, testing how well models integrate multi-source sensor data (drone trajectories and loop detectors) to predict future traffic states.

## Datasets

- **SimBarca** — total 101; splits: train (75), test (26)

## Metrics

- `MAE` **(primary)** — range: other
  - Mean Absolute Error: average of absolute differences between predicted and true speeds across all time steps and segments. Formula: MAE = (1/n) Σ |ŷ_i - y_i|.
- `RMSE` — range: other
  - Root Mean Square Error: square root of the average of squared differences between predicted and true speeds. Formula: RMSE = sqrt((1/n) Σ (ŷ_i - y_i)^2).
- `MAPE*` — range: percent
  - Modified Mean Absolute Percentage Error: average of absolute percentage errors, computed only where true speed > 1 m/s to avoid division by zero. Formula: MAPE* = (1/n) Σ |(ŷ_i - y_i)/y_i| for y_i > 1 m/s.

## Input / output format

**Input**: Time-series sequences of road segment speeds (from drones, sampled every 5s) and point speeds (from loop detectors, aggregated to 3-min intervals) over a 30-minute historical window, combined with an undirected graph representation of the road network (1570 nodes, 2803 edges).

**Output**: Predicted future traffic speeds (in m/s) for specific road segments and aggregated spatial regions over 15-minute or 30-minute horizons (5 or 10 time steps).

## Scoring recipe

```python
def compute_metrics(pred, true):
    # pred, true: arrays of shape (num_segments, num_steps)
    # Metrics are calculated per segment/region, then averaged
    mae = np.mean(np.abs(pred - true))
    rmse = np.sqrt(np.mean((pred - true)**2))
    mask = true > 1.0
    mape_star = np.mean(np.abs((pred[mask] - true[mask]) / true[mask])) * 100
    return mae, rmse, mape_star
```

## Common pitfalls

- MAPE becomes infinite when true speed is 0; the paper explicitly uses MAPE* by filtering out samples where speed ≤ 1 m/s.
- Loop detector data (point speed) systematically overestimates true segment speed due to sensor placement relative to stop lines, which can bias single-modality baselines if not accounted for.
- Prediction horizons are fixed at 15 and 30 minutes, corresponding to 5 and 10 discrete time steps in the dataset, not continuous time intervals.

## Evidence (verbatim from paper)

> Following the common practice in traffic forecasting literature[[8]], the prediction results are evaluated with three metrics: MAE, Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE). Their vector forms for a pair of prediction ($\hat{\mathbf{y}}$) and ground truth ($\mathbf{y}$) are defined as follows: [formula] where $n$ is an index for flattened predictions and labels. In the evaluation, the speed values take the unit of m/s, and the metrics are calculated for each road segment (or region) and then averaged over all segments (regions). Since a zero segment speed in the label will result in infinite MAPE, we only evaluate MAPE when the speed value is greater than 1 m/s, and we refer to this modified metric as MAPE*.

## Citation

```bibtex
@misc{xiong2025simbarca,
  title={Multi-Source Urban Traffic Flow Forecasting with Drone and Loop Detector Data},
  author={Xiong et al. (2025)},
  year={2025},
  note={arXiv:2501.03492}
}
```

- arXiv: 2501.03492