Stg4traffic Eval

stg4traffic-eval

STG4Traffic: A Survey and Benchmark of Spatial-Temporal Graph Neural Networks for Traffic Prediction — Luo et al. (2023) (arXiv:2307.00495, 2023)

What this evaluates

Evaluates the multi-step spatio-temporal forecasting capability of graph neural networks on urban traffic speed and flow prediction tasks. It measures how well models capture spatial dependencies and temporal dynamics across varying prediction horizons (3, 6, and 12 steps).

Datasets

• METR-LA — total 34272; splits: train (-1), val (-1), test (-1)
• PEMS-BAY — total 52116; splits: train (-1), val (-1), test (-1)
• PEMSD4 — total 16992; splits: train (-1), val (-1), test (-1)
• PEMSD8 — total 17856; splits: train (-1), val (-1), test (-1)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: average absolute difference between predicted and ground truth values. Formula: MAE = (1/M) * Σ|Yi - Ŷi|.
• RMSE — range: other
  • Root Mean Square Error: square root of the average squared difference between predicted and ground truth values. Formula: RMSE = √((1/M) * Σ(Yi - Ŷi)²).
• MAPE — range: percent
  • Mean Absolute Percentage Error: average absolute percentage difference between predicted and ground truth values. Formula: MAPE = (100%/M) * Σ|Yi - Ŷi|/Yi.

Input / output format

Input: Standardized (Z-Score) spatio-temporal graph sequences of traffic speed or flow sampled at 5-minute intervals. Historical window P=12 steps.

Output: Predicted traffic speed or flow values for the next Q=12 time steps.

Scoring recipe

def compute_metrics(y_true, y_pred):
    # Filter out zero values (noisy data) as per paper
    mask = y_true != 0
    y_t = y_true[mask]
    y_p = y_pred[mask]
    m = len(y_t)
    mae = np.mean(np.abs(y_t - y_p))
    rmse = np.sqrt(np.mean((y_t - y_p) ** 2))
    mape = np.mean(np.abs((y_t - y_p) / y_t)) * 100
    return mae, rmse, mape

Common pitfalls

• Zero values in traffic data represent missing or noisy readings and must be excluded from metric calculation; including them artificially inflates errors.
• Speed datasets (METR-LA, PEMS-BAY) use a 7:1:2 train/val/test split, while flow datasets (PEMSD4, PEMSD8) use a 6:2:2 split. Comparing results across dataset types requires accounting for this difference.
• Multi-step prediction horizons (3, 6, 12) exhibit error accumulation; models must be evaluated at each horizon separately rather than averaging across them.

Evidence (verbatim from paper)

In our experiments, we evaluate the model results using the Mask-Based Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) as metrics, where the zero values (i.e., noisy data) will be ignored [27]. Their initially defined equations are as follows: $$ \mathrm{MAE} = \frac{1}{M}\sum_{i = 1}^{M}|Y_{i} - \widehat{Y}{i}|, \mathrm{RMSE} = \sqrt{\frac{1}{M}\sum{i = 1}^{M}(Y_{i} - \widehat{Y}{i})^{2}}, \mathrm{MAPE} = \frac{100%}{M}\sum{i = 1}^{M}\left|\frac{Y_{i} - \widehat{Y}{i}}{Y{i}}\right|. \tag{20} $$ where $M$ is the number of values to predict, $Y_{i}$ is the prediction result and $\widehat{Y}_i$ is the ground truth. The smaller their values, the better the performance of the method is indicated.

Citation

@misc{luo2023stg4traffic,
  title={STG4Traffic: A Survey and Benchmark of Spatial-Temporal Graph Neural Networks for Traffic Prediction},
  author={Luo et al. (2023)},
  year={2023},
  note={arXiv:2307.00495}
}

• arXiv: 2307.00495