Road Safety Accident Eval

road-safety-accident-eval

Graph Neural Networks for Road Safety Modeling: Datasets and Evaluations for Accident Analysis — Nippani et al. (2023) (arXiv:2311.00164, 2023)

What this evaluates

Evaluates graph neural networks and embedding methods for predicting traffic accident occurrences and counts on road network edges. It probes the models' ability to capture spatial-temporal dependencies, leverage graph structural features, and benefit from multitask or transfer learning across different U.S. states.

Datasets

• Traffic Accident Dataset — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/VirtuosoResearch/ML4RoadSafety

Metrics

• MAE (primary) — range: count
  • Mean Absolute Error: the average of the absolute differences between predicted and actual accident counts per road segment. Formula: MAE = (1/N) * Σ|y_pred - y_true|.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve: measures the model's ability to distinguish between accident and non-accident edges across all classification thresholds.

Input / output format

Input: Graph-structured data where each instance is a road segment (edge) represented by concatenated node embeddings (degree, betweenness centrality, weather) and edge-level features (e.g., traffic volume/AADT). The model receives the full road network graph with temporal features.

Output: Predicted monthly accident count (regression) or binary accident occurrence label (classification) for each road segment.

Scoring recipe

def compute_metrics(y_true, y_pred, task='regression'):
    if task == 'regression':
        mae = np.mean(np.abs(y_true - y_pred))
        return {'MAE': mae}
    elif task == 'classification':
        auc = roc_auc_score(y_true, y_pred)
        return {'AUROC': auc}

Common pitfalls

• The dataset uses a chronological split (past years for training, future years for testing) rather than random shuffling, which is critical for temporal generalization.
• The positive label rate is extremely low (~0.2% or less), meaning models are evaluated on highly imbalanced data where complex spatiotemporal GNNs may underperform simpler ones due to insufficient training labels.
• Predictions are aggregated monthly from daily occurrences; evaluating at the wrong temporal granularity may misalign with the reported metrics.

Evidence (verbatim from paper)

For regression, we measure the mean absolute error (MAE) between the predicted number of accidents and the actual number of occurrences on a particular road. For classification, we measure AUROC scores.

Citation

@misc{nippani2023graph,
  title={Graph Neural Networks for Road Safety Modeling: Datasets and Evaluations for Accident Analysis},
  author={Nippani et al. (2023)},
  year={2023},
  note={arXiv:2311.00164}
}

• arXiv: 2311.00164