Seephys Eval

seephys-eval

SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning — Xiang et al. (2025) (arXiv:2505.19099, 2025)

What this evaluates

This benchmark evaluates multimodal LLMs' ability to perform physics reasoning using visual diagrams, text, or both. It probes visual interpretation, diagram-to-reasoning mapping, and the model's reliance on textual shortcuts versus actual visual perception across varying knowledge levels and diagram types.

Datasets

• SeePhys — total 2000; splits: test (2000)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Evaluated via a composite judgment strategy combining template matching and LLM-based extraction/judging (using DeepSeek-V3). SymPy is first used to screen straightforward answers; remaining responses are compared to ground truth using the LLM judge.

Input / output format

Input: Question text paired with physics diagrams (Text+Vision), question text with diagram captions (Text+Caption), question text only (Text Only), or a composite image rendered from question text and diagrams (Vision Only).

Output: A reasoning-augmented response containing significant figures, followed by a final answer extracted via template matching and LLM processing.

Scoring recipe

def score(response, gold):
    extracted = extract_answer_llm_or_template(response)
    if sympy_check(extracted, gold):
        return 1.0
    judge_prompt = f"Compare extracted answer '{extracted}' with ground truth '{gold}'."
    judge_output = llm_call(judge_prompt, model="DeepSeek-V3")
    return 1.0 if judge_output == "correct" else 0.0

accuracy = mean(score(r, g) for r, g in zip(predictions, golds))

Common pitfalls

• Models may achieve high accuracy in Text-Only settings for vision-optional problems by relying on textual shortcuts rather than visual reasoning, masking true multimodal capabilities.
• The composite judgment strategy uses an LLM (DeepSeek-V3) as the judge, which may introduce bias or inconsistency compared to strict template matching, especially for open-ended physics problems.
• Performance varies significantly across diagram types (e.g., Wave Motion, Circuit Diagram) and knowledge levels; reporting only aggregate accuracy masks domain-specific weaknesses.

Evidence (verbatim from paper)

We apply accuracy as the metric for this deterministic evaluation. In the experiments in this paper, we use DeepSeek-V3 as the extraction and judge model.

Citation

@misc{xiang2025seephys,
  title={SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning},
  author={Xiang et al. (2025)},
  year={2025},
  note={arXiv:2505.19099}
}

• arXiv: 2505.19099