Medirl Driver Attention Eval

medirl-driver-attention-eval

MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning — Baee et al. (2019) (arXiv:1912.07773, 2019)

What this evaluates

Predicts driver visual attention (fixation locations) in critical driving scenarios by modeling goal-directed attention sequences. It evaluates how well a model's predicted saliency map matches human gaze patterns across spatial and spatiotemporal cues.

Datasets

• DR(eye)VE — total 400; splits: train (280), val (40), test (80)
• BDD-A — total 1350; splits: train (945), val (135), test (270)
• DADA-2000 — total 534; splits: train (374), val (53), test (107)
• EyeCar — total ?; splits: test (-1); repo https://github.com/soniabaee/MEDIRL-EyeCar

Metrics

• CC (primary) — range: [0, 1]
  • Pearson Correlation Coefficient between flattened predicted and ground truth saliency maps. Higher values indicate better alignment with human fixation distributions.
• KLD — range: other
  • Kullback-Leibler Divergence between predicted and ground truth fixation distributions. Lower values indicate closer match to human attention patterns.
• s-AUC — range: [0, 1]
  • shuffled Area under the ROC curve, computed by comparing fixation locations against a shuffled background of non-fixated pixels. Higher values indicate better discrimination, penalizing central prediction bias.

Input / output format

Input: Sequences of 6 video frames resized to 144×256 and normalized. Frames are encoded using pre-trained backbones (HRNetV2, MaskTrack-RCNN, Monodepth2, VPGNet) to extract visual features including target, non-target, distance, lane, lead vehicle, driving task, and speed.

Output: A spatial probability map (saliency map) over the frame, generated by 7 softmax units representing fixation likelihood.

Scoring recipe

def compute_metrics(pred_map, gt_map):
    pred = pred_map.flatten()
    gt = gt_map.flatten()
    cc = pearsonr(pred, gt)
    kld = kl_divergence(gt, pred)
    s_auc = shuffled_auc(gt, pred)
    return {'CC': cc, 'KLD': kld, 's-AUC': s_auc}

Common pitfalls

• Driver attention naturally exhibits a strong center bias; models must be evaluated with s-AUC to avoid inflated scores from predicting central fixations.
• Training data is heavily filtered to exclude frames with >40% focus on irrelevant objects and to ensure KLD > 0.89 from average maps, which may limit generalization to routine driving.
• Metrics are computed on event-level sequences rather than isolated frames, requiring temporal consistency in predictions.

Evidence (verbatim from paper)

To evaluate attention prediction, we use location-based and distribution-based saliency metrics: KLD, shuffled Area under the ROC curve (s-AUC), and Correlation Coefficient (CC). We report s-AUC since it penalizes models with more central prediction.

Citation

@misc{baee2019medirl,
  title={MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning},
  author={Baee et al. (2019)},
  year={2019},
  note={arXiv:1912.07773}
}

• arXiv: 1912.07773