Vaexbench Eval

vaexbench-eval

Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence — Bang et al. (2026) (arXiv:2603.13091, 2026)

What this evaluates

Evaluates multimodal large language models' ability to perform extractive and abstractive spatiotemporal reasoning on egocentric videos. It probes long-horizon memory, object tracking, spatial orientation, and metric distance estimation under both multiple-choice and free-form generation settings.

Datasets

• VAEX-Bench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correct predictions out of total queries. For stochastic decoding stability, Accuracy@k is reported as the average accuracy across k independent generations per query (k=5 in this work). Free-form responses are scored via exact match or LLM-as-a-judge.

Input / output format

Input: Egocentric video (uniformly sampled to 32 frames for most models) paired with a fixed instruction header prompt. MCQ tasks include answer options; free-form tasks provide open-ended queries without options.

Output: Single selected option for MCQ tasks, or free-form text generation for open-ended queries.

Scoring recipe

def compute_accuracy(predictions, golds, task_type='mcq'):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if task_type == 'mcq':
            if pred.strip() == gold.strip():
                correct += 1
        elif task_type == 'freeform':
            if gold == 'Memory-Action':
                correct += llm_judge_score(pred, gold)
            else:
                if pred.strip() == gold.strip():
                    correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• MCQ evaluation may overestimate model capabilities due to option elimination or chance selection; free-form generation reveals a significant performance drop.
• Frame sampling is not uniformly controllable across all model APIs (e.g., Gemini), potentially introducing input distribution shifts that affect temporal reasoning.
• Averaging Accuracy@5 across 5 generations may smooth out catastrophic failures in long-horizon reasoning tasks.

Evidence (verbatim from paper)

We evaluate in a zero-shot setting using each MLLM’s default interface. Following prior work in MLLM evaluation, we adopt a fixed decoding configuration across MLLMs with temperature 0.7, top-p =1.0, and top-k =40. We evaluate stochastic decoding stability by reporting Accuracy@k, defined as the average accuracy across k independent generations per query. In our experiments, we set k=5 and report Accuracy@5 for both MCQ and numerical-answer items.

Citation

@misc{bang2026vaexbench,
  title={Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence},
  author={Bang et al. (2026)},
  year={2026},
  note={arXiv:2603.13091}
}

• arXiv: 2603.13091