Humanvbench Eval

humanvbench-eval

HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data — Ting Zhou et al. (2024) (arXiv:2412.17574, 2024)

What this evaluates

This benchmark evaluates the human-centric video understanding capabilities of multimodal large language models (MLLMs). It specifically probes inner emotion perception, outer behavioral manifestations, and cross-modal speech-visual alignment through 16 fine-grained multiple-choice tasks.

Datasets

• HumanVBench — total ?; splits: test (-1); repo https://github.com/modelscope/data-juicer

Metrics

• accuracy (primary) — range: percent
  • Calculated as the proportion of correctly answered multiple-choice questions out of the total number of questions. Reported alongside random guess performance and graduate-level human baselines for reference.

Input / output format

Input: Video clips (with or without audio) accompanied by a multiple-choice question (N choose 1) probing human-centric attributes like emotion, behavior, or speech-visual alignment.

Output: A single selected option from the provided multiple-choice options.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Fixed 8-frame sampling introduces temporal noise, causing models to misclassify neutral or specific emotions (e.g., interpreting mouth-opening frames as surprise).
• Visual-only MLLMs lack audio input, causing them to degrade to speech action recognition rather than performing true lip-reading or speech-visual alignment.
• Time-specific tasks require explicit timestamp integration in prompts, but many models lack native support, leading to poor temporal reasoning without manual intervention.

Evidence (verbatim from paper)

All QAs were framed as multiple-choice questions (N choose 1, with N varying across different test samples), reporting both accuracy and the performance of random guesses and graduate-level humans for reference.

Citation

@misc{zhou2024humanvbench,
  title={HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data},
  author={Ting Zhou et al. (2024)},
  year={2024},
  note={arXiv:2412.17574}
}

• arXiv: 2412.17574