Perceptioncomp Eval

perceptioncomp-eval

PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning — Li et al. (2026) (arXiv:2603.26653, 2026)

What this evaluates

This benchmark evaluates long-horizon, perception-centric video reasoning in multimodal LLMs. It requires models to gather visual evidence across temporally separated segments and integrate multiple compositional constraints (e.g., object recognition, temporal tracking, spatial inference) to answer complex questions.

Datasets

• PerceptionComp — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total number of samples) * 100%.

Input / output format

Input: Raw videos for native models, or uniformly sampled frames (typically 64, sometimes 50) for non-native models, paired with multiple-choice or open-ended questions requiring multi-step reasoning.

Output: Direct answer choice for instruction-tuned models, or Chain-of-Thought reasoning traces followed by the final answer for thinking-style models.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred.strip() == gold.strip())
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models often correctly identify objects or moments but fail on fine-grained attributes or spatial relations, causing mid-chain reasoning collapses.
• Performance is highly sensitive to input frame density and thinking-token budget, so results vary significantly based on sampling and generation limits.
• Single-view human accuracy is near chance (18.97%), proving that language priors or single-frame inspection are insufficient.

Evidence (verbatim from paper)

We report comprehensive results in Table[2]. Most models achieve accuracy below 40%, indicating that PerceptionComp is challenging for current video MLLMs.

Citation

@misc{li2026perceptioncomp,
  title={PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2603.26653}
}

• arXiv: 2603.26653