Mmhu Eval

mmhu-eval

MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding — Li et al. (2025) (arXiv:2507.12463, 2025)

What this evaluates

Evaluates multimodal models on human behavior understanding in autonomous driving contexts, covering motion prediction, text-to-motion generation, and behavior question-answering. It probes a model's ability to interpret video frames, predict future human motion, generate plausible driving-scene motions from text, and answer safety-critical behavior questions.

Datasets

• MMHU — total 57340; splits: train (56500), test (840)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered close-ended questions about human behaviors in driving scenes. Computed as correct predictions divided by total test instances.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for multi-class behavior classification. Balances false positives and false negatives in VQA and intention prediction tasks.
• MPJPE — range: other
  • Mean Per Joint Position Error. Average Euclidean distance between predicted and ground-truth 3D joint coordinates across all joints and frames.
• FID — range: other
  • Fréchet Inception Distance. Measures the Wasserstein-2 distance between feature distributions of real and generated motion sequences. Lower values indicate better generation quality.

Input / output format

Input: Multimodal input consisting of 4 to 6 sampled video frames and a text prompt or close-ended question describing a specific human behavior or motion trajectory.

Output: Text answer for VQA tasks; 3D/2D joint coordinates or motion sequences for prediction/generation tasks.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return correct / len(gold)

def compute_f1(predictions, gold, labels):
    tp = sum(1 for p, g in zip(predictions, gold) if p == g)
    prec = tp / max(1, sum(1 for p in predictions if p in labels))
    rec = tp / max(1, sum(1 for g in gold if g in labels))
    return 2 * prec * rec / max(1e-9, prec + rec)

def compute_mpjpe(pred_joints, gt_joints):
    return np.mean(np.sqrt(np.sum((pred_joints - gt_joints)**2, axis=-1)))

Common pitfalls

• Models are evaluated on different test sets depending on the task: MMHU-T for VQA/motion, 3DPW for motion prediction generalization, and JAAD for intention prediction.
• VQA baselines use only 4-6 frames per query, which may underrepresent temporal context compared to full video inputs.
• Motion generation metrics (FID) are computed on generic datasets but applied to driving scenes, highlighting a domain gap rather than absolute performance.

Evidence (verbatim from paper)

We evaluated the pre-trained baselines on MMHU-T, to unify the time argument used in computing the MPJPE metric, we use the frame id to replace it. All baselines are evaluated on the same frames.

Citation

@misc{li2025mmhu,
  title={MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2507.12463}
}

• arXiv: 2507.12463