qhjqhj00 video-phy-2-eval
VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation — Bansal et al. (2025) (arXiv:2503.06800, 2025)
What this evaluates
Evaluates the ability of text-to-video models to generate physically plausible content by testing adherence to real-world action-centric physical rules, such as conservation of mass/momentum and gravity. It probes whether models understand fundamental physical commonsense beyond superficial motion or visual aesthetics.
Datasets
- VideoPhy-2 — total 6800; splits: full (-1), hard (-1), PA (-1), OI (-1)
Metrics
joint_performance(primary) — range: percent- Percentage of generated videos that correctly satisfy both the text prompt constraints and the associated physical rules, as determined by human annotators or the VideoPhy-2-Autoeval rater.
violation_score— range: percent- Calculated as the fraction of associated physical rules that are violated in a generated video: (number of violated rules) / (total associated rules) * 100.
semantic_adherence— range: correlation- Correlation score measuring how well the generated video aligns with the input text prompt, primarily used for auto-rater validation.
physical_commonsense_score— range: correlation- Correlation score measuring alignment with physical rule judgments, primarily used for auto-rater validation.
Input / output format
Input: Text prompt describing a real-world action or physical scenario.
Output: Generated video file corresponding to the input prompt.
Scoring recipe
def score_video(video, prompt, gold_rules):
# Joint performance: pass/fail based on human/auto judgment
joint_pass = judge_adherence(video, prompt) and judge_physics(video, gold_rules)
joint_performance = 1.0 if joint_pass else 0.0
# Violation score
violated = [rule for rule in gold_rules if check_violation(video, rule)]
violation_score = (len(violated) / len(gold_rules)) * 100 if gold_rules else 0.0
return joint_performance, violation_score
Common pitfalls
- Assuming that optimizing for visual aesthetics or motion smoothness (e.g., via optical flow) will improve physical rule adherence, as the paper explicitly shows these metrics are poorly correlated.
- Treating all dataset splits equally; the hard split yields significantly lower scores (
22%) than the full dataset (33%), which can mask model limitations if not reported separately. - Overgeneralizing closed vs. open model performance; closed models do not inherently outperform open ones on physical commonsense tasks, as demonstrated by Ray2 underperforming Wan2.1-14B.
Evidence (verbatim from paper)
We compare the joint performance of various open and closed text-to-video generative models on the VideoPhy-2 dataset in Table [2]. Specifically, we present their performance on the entire dataset, the hard split, and subsets focused on physical activities/sports (PA) and object interactions (OI). Even the best-performing model, Wan2.1-14B, achieves only 32.6% and 21.9% on the full and hard splits of our dataset, respectively. For example, if 100 physical rules are associated with the law of gravity and 25 of them are violated, the violation score would be 25%.
Citation
@misc{bansal2025videophy2,
title={VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation},
author={Bansal et al. (2025)},
year={2025},
note={arXiv:2503.06800}
}
- arXiv: 2503.06800