Phycritic Eval

phycritic-eval

PhyCritic: Multimodal Critic Models for Physical AI — Xiong et al. (2026) (arXiv:2602.11124, 2026)

What this evaluates

Evaluates multimodal models' ability to perform physical reasoning, spatial cognition, and egocentric task planning, as well as their capacity to act as reliable critics/judges for physical AI tasks.

Datasets

• PhyCritic-Bench — total ?; splits: test (-1)
• VL-RewardBench — total ?; splits: test (-1)
• Multimodal-RewardBench — total ?; splits: test (-1)
• CosmosReason1-Bench — total ?; splits: test (-1)
• CV-Bench — total ?; splits: test (-1)
• EgoPlanBench2 — total ?; splits: test (-1)

Metrics

• accuracy (overall/macro) (primary) — range: percent
  • Percentage of correctly answered questions or correctly judged critiques across the benchmark. Reported as overall accuracy and macro-averaged across sub-tasks.
• 2D/3D spatial score — range: percent
  • Accuracy on 2D and 3D spatial cognition sub-tasks within CV-Bench.

Input / output format

Input: Multimodal prompts consisting of images or video frames paired with text questions or instructions for physical reasoning, spatial cognition, egocentric planning, or critique/judgment tasks.

Output: Textual predictions, answers, or structured critique/judgment outputs. For critic benchmarks, the model generates its own physics-aware prediction before evaluating candidate responses.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if normalize(pred) == normalize(gold))
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Requires ground-truth answers for the self-prediction reward during training, limiting applicability to fully open-ended scenarios.
• Benchmarks focus on curated physical QA and specific sub-tasks, which may not generalize to unstructured real-world embodied settings.
• Macro-averaged scores can mask performance drops on specific sub-domains like bridge-v2 or failure-robustness.

Evidence (verbatim from paper)

On PhyCritic-Bench (Tab. [2]), our model attains the best overall accuracy among open-source 7B/8B models (68.0)... On CosmosReason1-Bench, it achieves the best open-source accuracy (63.9)... We evaluate our approach on both critic performance and its capacity for handling Physical-AI tasks. For critic performance, we use our curated PhyCritic-Bench for physical-related critique, together with two general reward benchmarks—VL-RewardBench and Multimodal-RewardBench... For physical capacity, we evaluate our methods on three physical-related multimodal benchmarks. CosmosReason1-Bench probes physical commonsense... CV-Bench assesses whether visual representations support downstream control... EgoPlanBench2 targets egocentric daily-task planning...

Citation

@misc{xiong2026phycritic,
  title={PhyCritic: Multimodal Critic Models for Physical AI},
  author={Xiong et al. (2026)},
  year={2026},
  note={arXiv:2602.11124}
}

• arXiv: 2602.11124