Madrid Traffic Forecasting Eval

madrid-traffic-forecasting-eval

Deep Echo State Networks for Short-Term Traffic Forecasting: Performance Comparison and Statistical Assessment — Del Ser et al. (2020) (arXiv:2004.08170, 2020)

What this evaluates

Evaluates the short-term traffic flow forecasting capability of various machine learning and deep learning models on real-world urban road networks. It specifically probes how well models generalize across different traffic profiles and prediction horizons while maintaining computational efficiency.

Datasets

• Madrid Traffic Dataset — total 133; splits: 10-fold time-split (-1)

Metrics

• R^2 (Coefficient of determination) (primary) — range: [-inf, 1]
  • Calculated as 1 - (sum((y_true - y_pred)^2) / sum((y_true - mean(y_true))^2)), measuring the proportion of variance in traffic flow explained by the model.

Input / output format

Input: Traffic flow time series aggregated every 15 minutes, transformed into supervised learning instances using a sliding window of W=6 past samples.

Output: Predicted traffic flow value for a given prediction horizon h ∈ {1, 2, 3, 4} slots.

Scoring recipe

def compute_r2(y_true, y_pred):
    ss_res = sum((y_true - y_pred) ** 2)
    ss_tot = sum((y_true - mean(y_true)) ** 2)
    return 1 - (ss_res / ss_tot)

# For each ATR and horizon h:
for a in range(133):
    scores = []
    for p in range(10):  # 10 time-split partitions
        train, test = split_time_series(data[a], p)
        model.fit(train)
        y_pred = model.predict(test)
        scores.append(compute_r2(test.targets, y_pred))
    avg_score[a] = mean(scores)

# Statistical ranking across M models:
# 1. Friedman test on avg_scores across models
# 2. If significant, Wilcoxon signed-rank test for pairwise comparisons
# 3. Count WINS, TIES, LOSSES per model
# 4. Compute fractional rankings based on WINS/TIES
# 5. Average ranks across all 133 ATRs
# 6. Compute Nemenyi Critical Distance: CD = Q * sqrt(M*(M+1)/(6*N_ATR))

Common pitfalls

• Using random k-fold cross-validation instead of chronological time-split splits, which leaks future information into training.
• Ignoring the sliding window transformation (W=6) when preparing input sequences, leading to shape mismatches.
• Applying standard accuracy metrics (e.g., MAE, RMSE) without converting to R^2 or failing to report statistical significance via Friedman/Wilcoxon tests.

Evidence (verbatim from paper)

Each hyper-parameter combination was evaluated over a separate subset of 10 ATRs in terms of their average coefficient of determination (R^2). ... Given a prediction horizon h and an ATR, R^2 scores were recorded for every model over 10 train/test time-split partitions... two statistical hypothesis tests were applied at a significance level alpha=0.05: 1. A Friedman test, to enforce an initial check whether statistically significant gaps are present... 2. A Wilcoxon signed rank test, to assess the relevance of the difference among means in pairwise comparisons...

Citation

@misc{delser2020deep,
  title={Deep Echo State Networks for Short-Term Traffic Forecasting: Performance Comparison and Statistical Assessment},
  author={Del Ser et al. (2020)},
  year={2020},
  note={arXiv:2004.08170}
}

• arXiv: 2004.08170