Gameplayqa Eval

gameplayqa-eval

GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents — Wang et al. (2026) (arXiv:2603.24329, 2026)

What this evaluates

GameplayQA evaluates multi-modal large language models' ability to understand decision-dense, first-person synchronized multi-video environments. It probes capabilities in agent-state tracking, temporal reasoning, and cross-video event alignment across three cognitive difficulty levels.

Datasets

• GameplayQA — total 2400; splits: test (2400)

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match accuracy for multiple-choice questions. Computed as the fraction of correctly answered questions after extracting the predicted option letter.

Input / output format

Input: A video clip (or sampled frames at 1 FPS, max 32 frames, resized to 720p longer side) paired with a multiple-choice question describing an event, state, or action in the video. Models are instructed to output a single letter corresponding to the correct option.

Output: Ideally a single letter (A, B, C, or D), but models frequently output full sentences or explanations. The final selected option is extracted using an LLM judge (GPT-5-mini).

Scoring recipe

def compute_accuracy(predictions, gold_options):
    extracted = [extract_letter(pred) for pred in predictions]  # via GPT-5-mini judge
    correct = sum(1 for e, g in zip(extracted, gold_options) if e == g)
    return correct / len(gold_options)

Common pitfalls

• Models often ignore the single-letter instruction and output full explanations, requiring an external LLM judge for option extraction.
• Frame sampling strategy (1 FPS, capped at 32 frames) heavily influences temporal reasoning performance; longer videos lose critical motion cues.
• Language priors can artificially inflate scores on degraded inputs (e.g., no video or random frames), masking true visual grounding capabilities.

Evidence (verbatim from paper)

We evaluate all models in a zero-shot setting using accuracy as the metric. For video-native models (Gemini, Seed), we input the entire video directly. For frame-based models, we sample frames at 1 FPS up to 32 frames; for videos longer than 32 seconds, we uniformly sample 32 frames across the duration. Videos are resized such that the longer side is 720p while preserving aspect ratio. Although models are instructed to output a single letter, they sometimes produce full sentences or explanations; we use GPT-5-mini as an LLM judge to extract the selected option.

Citation

@misc{wang2026gameplayqa,
  title={GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2603.24329}
}

• arXiv: 2603.24329