Driveact Eval

driveact-eval

Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition — Tianyi Liu et al. (arXiv:2604.05947, 2026)

What this evaluates

Evaluates fine-grained driver action recognition in constrained in-cabin environments using multimodal video inputs (RGB, IR, Depth). It probes the model's ability to classify 34 specific driver activities under variable illumination and occlusion by measuring both overall and per-class recognition accuracy.

Datasets

• Drive&Act — total ?; splits: Split 0 (-1), Split 1 (-1), Split 2 (-1)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Measures the proportion of correct top-ranked predictions across all test instances.
• Mean-1 accuracy — range: percent
  • Calculates the average accuracy across all 34 fine-grained activity classes, mitigating the effect of class imbalance.

Input / output format

Input: Multimodal video sequences (RGB, IR, Depth) sampled at 8 frames per video, resized to 224×224 pixels, processed through a video transformer backbone to yield 1568 768-dimensional spatio-temporal tokens.

Output: Classification probability distribution over 34 fine-grained driver activity classes; final prediction is the class with the highest probability.

Scoring recipe

def compute_metrics(predictions, labels):
    top1_correct = sum(p == l for p, l in zip(predictions, labels))
    top1_acc = (top1_correct / len(labels)) * 100
    class_accuracies = []
    for c in range(34):
        class_mask = [l == c for l in labels]
        if sum(class_mask) > 0:
            correct = sum(1 for p, l in zip(predictions, labels) if l == c and p == c)
            total = sum(class_mask)
            class_accuracies.append(correct / total)
    mean1_acc = (sum(class_accuracies) / len(class_accuracies)) * 100
    return top1_acc, mean1_acc

Common pitfalls

• Results must be averaged across the three predefined dataset splits; reporting a single split's score misrepresents the official benchmark result.
• Mean-1 accuracy is the average of per-class accuracies, not the overall accuracy; it specifically addresses class imbalance in fine-grained driver actions.
• The evaluation strictly uses RGB, IR, and Depth modalities from the right-top camera view, ignoring NIR and 3D skeleton modalities present in the full dataset.

Evidence (verbatim from paper)

We follow the three predefined splits provided by the data set for a consistent evaluation and average the results across these splits. For performance evaluation, we use two metrics: Top-1 accuracy (Top-1 Acc.) and Mean-1 accuracy (Mean-1 Acc.). Top-1 Acc. measures the proportion of correct top-ranked predictions, while Mean-1 Acc. calculates the average accuracy across all classes, mitigating the effect of class imbalance.

Citation

@misc{liu2026driveact,
  title={Mixture-of-Modality-Experts with Holistic Token Learning for Fine-Grained Multimodal Visual Analytics in Driver Action Recognition},
  author={Tianyi Liu et al.},
  year={2026},
  note={arXiv:2604.05947}
}

• arXiv: 2604.05947