Injury Severity Prediction Eval

injury-severity-prediction-eval

Predicting Seriousness of Injury in a Traffic Accident: A New Imbalanced Dataset and Benchmark — Lagias et al. (2022) (arXiv:2205.10441, 2022)

What this evaluates

Evaluates a model's ability to predict traffic accident injury severity (fatal, serious, slight) from tabular accident data. It specifically probes performance on highly imbalanced multi-class classification, focusing on minority-class accuracy and the impact of data imputation and resampling techniques.

Datasets

• UK DfT Traffic Accident Dataset (2005–2019) — total 564000; splits: train (-1), val (-1), test (-1); repo https://ale66.github.io/traffic-accident-gravity-predictor/

Metrics

• Overall classification accuracy (primary) — range: [0, 1]
  • Correctly classified test samples divided by the total number of test samples.
• Class accuracy — range: [0, 1]
  • Accuracy computed independently for each injury severity class (fatal, serious, slight) by dividing correct predictions for that class by the total number of instances belonging to that class in the test set.

Input / output format

Input: Tabular traffic accident records containing features from DfT data (e.g., vehicle type, speed, road conditions, casualty details).

Output: Predicted injury severity class: 'fatal', 'serious', or 'slight'.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_class_accuracy(predictions, gold, target_class):
    mask = [g == target_class for g in gold]
    if not any(mask): return 0.0
    correct = sum(1 for p, g in zip(predictions, gold) if p == g and g == target_class)
    return correct / sum(mask)

Common pitfalls

• The dataset is highly imbalanced with 'fatal' injuries being the rarest class, leading to poor baseline accuracy (~25-53%) without careful class weighting or resampling.
• Applying SMOTE or MissForest imputation does not consistently improve minority-class performance and may require extensive hyperparameter tuning to yield benefits.
• Reinforcement learning models are prone to overestimation when trained for too many episodes, requiring careful reward shaping and episode limits.

Evidence (verbatim from paper)

In training and validation, 2005–2018 accident data (cf. with Table 1) were used with a 75%–25% split. Different test sets were used, as described below, and the following metrics: Overall classification accuracy: correctly classified test samples over total number of test samples; Class accuracy: performance in testing on each single class.

Citation

@misc{lagias2022predicting,
  title={Predicting Seriousness of Injury in a Traffic Accident: A New Imbalanced Dataset and Benchmark},
  author={Lagias et al. (2022)},
  year={2022},
  note={arXiv:2205.10441}
}

• arXiv: 2205.10441