Physgame Eval

physgame-eval

PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos — Cao et al. (2024) (arXiv:2412.01800, 2024)

What this evaluates

This benchmark evaluates a model's ability to reason about physical laws and detect physical commonsense violations in gameplay videos. It probes spatial, temporal, and meta-information-based physical reasoning through curated multi-choice questions.

Datasets

• PhysGame — total ?; splits: test (-1); repo https://github.com/PhysGame/PhysGame

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multi-choice questions. Average accuracy is computed as the mean accuracy across all 12 evaluated physical domains (e.g., friction, gravity, elasticity, reflection).

Input / output format

Input: Video clips from gameplay footage (frame count and extraction config varies per model, following Video-MME official settings) paired with a multi-choice question asking to identify physical commonsense violations or reason about physical properties.

Output: A single selected option (e.g., A, B, C, or D) corresponding to the correct answer for the multi-choice question.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    total = len(gold)
    return (correct / total) * 100 if total > 0 else 0.0

# Average accuracy across domains:
# avg_acc = np.mean([compute_accuracy(preds[d], gold[d]) for d in domains])

Common pitfalls

• Frame extraction and configuration must strictly follow each model's official Video-MME settings rather than using a fixed number of frames.
• The evaluation prompt is not in the main text but located in the supplementary material, which can lead to inconsistent prompting if overlooked.
• Accuracy is reported per physical domain and as an overall average; reporting only the overall average without domain breakdown loses granularity.

Evidence (verbatim from paper)

We employ accuracy as the evaluation metric for our curated multi-choice questions. The evaluation results on the PhysGame benchmark are demonstrated in Table [4]. Among all proprietary models, GPT-4o and Gemini-1.5-pro demonstrate the best performance, achieving average accuracy scores of 56.1% and 55.2%, respectively.

Citation

@misc{cao2024physgame,
  title={PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos},
  author={Cao et al. (2024)},
  year={2024},
  note={arXiv:2412.01800}
}

• arXiv: 2412.01800