Perception Test Eval

perception-test-eval

Perception Test: A Diagnostic Benchmark for Multimodal Video Models — Pătrăucean et al. (2023) (arXiv:2305.13786, 2023)

What this evaluates

Evaluates multimodal video models on core perception skills and reasoning types across six computational tasks, including tracking, temporal localization, and video question answering. It probes zero-shot and few-shot generalization on real-world videos with dense annotations.

Datasets

• Perception Test — total 11600; splits: validation (-1); repo https://github.com/deepmind/perception_test

Metrics

• top-1 accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted answers out of the total number of multiple-choice questions.
• Avg. IoU — range: [0, 1]
  • Average Intersection over Union between predicted and ground-truth bounding box trajectories.
• Avg. Jaccard — range: [0, 1]
  • Average Jaccard index for point tracking trajectories.
• mAP — range: [0, 100]
  • mean Average Precision for temporal action or sound localization segments.
• HOTA — range: [0, 1]
  • Higher Order Tracking Accuracy for grounded video question answering bounding box tracks.

Input / output format

Input: Video with audio, plus task-specific instructions (e.g., bounding box coordinates for tracking, or a natural language question with multiple-choice options for videoQA).

Output: Task-specific predictions: bounding box or point trajectories, lists of temporal segments, or a single selected answer choice.

Scoring recipe

def compute_top1_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

# For tracking/localization, compute per-frame IoU/Jaccard/mAP
# and average across the dataset as specified in Table 4.

Common pitfalls

• Models often fail on counterfactual questions by latching onto visible video elements instead of imagining alternate realities.
• Fine-tuning is currently required for action and sound localization tasks because open-vocabulary models do not yet exist, violating the intended zero-shot evaluation setting.
• Hard negative options and adversarial actions in the dataset cause significant performance drops compared to standard benchmarks like NExT-QA.

Evidence (verbatim from paper)

For all the tasks, the video and audio are available as inputs, together with a task specification where applicable, e.g. the coordinates of a box to track for object tracking, or a language question and options for multiple-choice videoQA. multiple-choice videoQA | answer (1 out of 3) | top-1 accuracy | SeViLA | 46.2

Citation

@misc{patraucean2023perceptiontest,
  title={Perception Test: A Diagnostic Benchmark for Multimodal Video Models},
  author={Pătrăucean et al. (2023)},
  year={2023},
  note={arXiv:2305.13786}
}

• arXiv: 2305.13786