Baton Eval

baton-eval

BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving — Wang et al. (2026) (arXiv:2604.07263, 2026)

What this evaluates

This benchmark evaluates a model's ability to understand coarse driving actions and predict bidirectional control transitions between human drivers and automated driving systems. It probes multimodal fusion capabilities by testing whether models can leverage synchronized video, vehicle telemetry, and route context to forecast handovers and takeovers under varying time horizons.

Datasets

• BATON — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/OpenLKA/BATON

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified driving actions out of total samples across seven coarse classes.
• Macro-F1 — range: [0, 1]
  • Unweighted mean of the F1-score computed independently for each of the seven driving action classes.
• AUROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates across classification thresholds.
• AUPRC (primary) — range: [0, 1]
  • Area under the Precision-Recall Curve, measuring the trade-off between precision and recall across thresholds, particularly sensitive to class imbalance in transition prediction.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for the binary transition prediction task.

Input / output format

Input: A synchronized 5-second multimodal observation window ending at time t, comprising front-view video, in-cabin video, CAN bus signals (vehicle state, planning, lead-vehicle interactions), and GPS route context.

Output: Task 1: One of seven discrete driving action classes (Cruising, Accelerating, Braking, Turning, Lane Change, Stopped, Car Following). Tasks 2 & 3: Binary probability or class indicating whether a control transition (Human→DA or DA→Human) will occur within a specified future horizon [t, t+h].

Scoring recipe

def compute_metrics(predictions, labels, task_id):
    if task_id == 1:
        acc = sum(p == l for p, l in zip(predictions, labels)) / len(labels)
        f1 = macro_f1_score(labels, predictions)
        return {'Accuracy': acc, 'Macro-F1': f1}
    else:
        auroc = roc_auc_score(labels, predictions)
        auprc = average_precision_score(labels, predictions)
        f1 = f1_score(labels, predictions)
        return {'AUROC': auroc, 'AUPRC': auprc, 'F1': f1}

Common pitfalls

• Using random or cross-vehicle splits instead of the mandated cross-driver split for primary benchmarking results.
• Failing to respect the 0.5s stride and 5s observation window, which causes temporal data leakage or incorrect sample extraction.
• Treating handover (Human→DA) and takeover (DA→Human) tasks identically without accounting for their asymmetric temporal dynamics and different positive sample construction windows.
• Relying solely on visual modalities, as the benchmark explicitly demonstrates that structured vehicle and route context are necessary for reliable transition prediction.

Evidence (verbatim from paper)

We report Accuracy and Macro-F1. ... We report AUROC, AUPRC (primary), and F1.

Citation

@misc{wang2026baton,
  title={BATON: A Multimodal Benchmark for Bidirectional Automation Transition Observation in Naturalistic Driving},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.07263}
}

• arXiv: 2604.07263