Phyx Eval

phyx-eval

PhyX: Does Your Model Have the "Wits" for Physical Reasoning? — Hui Shen et al. (2025) (arXiv:2505.15929, 2025)

What this evaluates

Probes multimodal physical reasoning by requiring models to interpret realistic visual scenarios, understand implicit physical conditions, and apply domain-specific knowledge across six physics domains. It evaluates both visual grounding and the ability to integrate symbolic reasoning with real-world constraints.

Datasets

• PhyX — total 18000; splits: testmini (6000)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered instances out of the total evaluated. Calculated as (number of correct predictions / total instances) * 100.

Input / output format

Input: An image depicting a realistic physical scenario, accompanied by a problem description (text) and a question. Text input varies across three settings: Full-Text, Text-DeRedundancy, and Text-Minimal.

Output: Raw text containing Chain-of-Thought reasoning steps followed by a definitive answer. For multiple-choice questions, the model outputs the option letter or answer text.

Scoring recipe

def compute_accuracy(predictions, gold_answers, question_types):
    correct = 0
    for pred, gold, q_type in zip(predictions, gold_answers, question_types):
        # Extract definitive answer from raw CoT output
        extracted = rule_based_extract(pred) if q_type == 'MC' else llm_judge_extract(pred)
        # Compare to ground truth
        if q_type == 'MC':
            is_correct = (extracted == gold) or llm_judge_match(extracted, gold)
        else:
            is_correct = llm_judge_correctness(extracted, gold)
        if is_correct:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Open-ended answers can be phrased in myriad ways, making exact string matching ineffective; an LLM judge is required for fair scoring.
• Multiple-choice formats allow models to exploit surface-level cues or eliminate options without genuine reasoning, underestimating true capability gaps.
• Heavy reliance on textual descriptions means performance drops sharply when text is minimized, masking visual reasoning deficits if only full-text prompts are used.

Evidence (verbatim from paper)

The quantitative evaluations in all subsequent experiments were assessed on this testmini subset. ... For OE questions, the next step is comparing the extracted answer against the ground truth to determine its correctness. ... For MC questions, we first attempt to directly match the option letter. If this direct matching fails, we then use a LLM as a judge... Notably, even worst human experts achieve accuracy of 75.6%, significantly outperforming all the models included in our comparative analysis.

Citation

@misc{shen2025phyx,
  title={PhyX: Does Your Model Have the "Wits" for Physical Reasoning?},
  author={Hui Shen et al. (2025)},
  year={2025},
  note={arXiv:2505.15929}
}

• arXiv: 2505.15929