Sphinx Eval

sphinx-eval

SPHINX: A Synthetic Environment for Visual Perception and Reasoning — Alam et al. (2025) (arXiv:2511.20814, 2025)

What this evaluates

Probes visual perception and reasoning capabilities of vision-language models across 25 distinct task types, including symmetry, spatial transformations, chart interpretation, and sequence prediction. Uses a synthetic environment with verifiable ground truth to measure model accuracy against human baselines.

Datasets

• Sphinx — total 2500; splits: test (2500); repo https://github.com/xashru/sphinx

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correctly answered questions out of the total number of questions. Answers are extracted using the mathruler library, with GPT-5-mini as a fallback for ambiguous extractions.

Input / output format

Input: An image and a natural language question describing a visual reasoning task (e.g., symmetry, spatial transformation, chart interpretation, or sequence prediction).

Output: A natural language answer or selected option, processed through the mathruler library for extraction and verification.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    extracted = mathruler_extract(pred)
    if extracted == gold:
        correct += 1
accuracy = correct / len(gold_answers)

Common pitfalls

• Answer extraction relies on an external library (mathruler) with an LLM fallback, which may introduce extraction bias or inconsistency across models.
• Models are evaluated using their 'default reasoning settings', which are not standardized or optimized, making cross-model comparisons sensitive to inference hyperparameters.
• Human performance is reported as a single aggregate gap (24.3%) without detailing the human evaluation protocol or task breakdown.

Evidence (verbatim from paper)

We curated the Sphinx benchmark to include 2,500 questions, with 100 instances per task. We evaluated three proprietary variants of GPT-5 (regular, mini, and nano) using their default reasoning settings*. In addition, we evaluated nine open-source vision-language models, including the Qwen2.5-VL family*, Llama 3.2*, InternVL3*, and the Qwen3-VL family*, covering parameter scales from 3B to 38B. The evaluation prompt is provided in the Appendix[G]. We use the mathruler library* for answer extraction and verification, with GPT-5-mini serving as a fallback when extraction is ambiguous. For open-source models, we use the VLMEvalKit framework for inference*. Results are summarized in Table[1]. Overall, GPT-5 achieves the best performance with an average accuracy of 51.1% on all tasks, although it still falls short of human accuracy by 24.3%.

Citation

@misc{alam2025sphinx,
  title={SPHINX: A Synthetic Environment for Visual Perception and Reasoning},
  author={Alam et al. (2025)},
  year={2025},
  note={arXiv:2511.20814}
}

• arXiv: 2511.20814