# Sphinx Eval

> Probes visual perception and reasoning capabilities of vision-language models across 25 distinct task types, including symmetry, spatial transformations, chart interpretation, and sequence prediction. Uses a synthetic environment with verifiable ground truth to measure model accuracy against human baselines. Use when the user wants to benchmark on Sphinx, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/sphinx-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/sphinx-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/sphinx-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/sphinx-eval

---


# sphinx-eval

> SPHINX: A Synthetic Environment for Visual Perception and Reasoning — Alam et al. (2025) (arXiv:2511.20814, 2025)

## What this evaluates

Probes visual perception and reasoning capabilities of vision-language models across 25 distinct task types, including symmetry, spatial transformations, chart interpretation, and sequence prediction. Uses a synthetic environment with verifiable ground truth to measure model accuracy against human baselines.

## Datasets

- **Sphinx** — total 2500; splits: test (2500); repo https://github.com/xashru/sphinx

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions out of the total number of questions. Answers are extracted using the mathruler library, with GPT-5-mini as a fallback for ambiguous extractions.

## Input / output format

**Input**: An image and a natural language question describing a visual reasoning task (e.g., symmetry, spatial transformation, chart interpretation, or sequence prediction).

**Output**: A natural language answer or selected option, processed through the mathruler library for extraction and verification.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    extracted = mathruler_extract(pred)
    if extracted == gold:
        correct += 1
accuracy = correct / len(gold_answers)
```

## Common pitfalls

- Answer extraction relies on an external library (mathruler) with an LLM fallback, which may introduce extraction bias or inconsistency across models.
- Models are evaluated using their 'default reasoning settings', which are not standardized or optimized, making cross-model comparisons sensitive to inference hyperparameters.
- Human performance is reported as a single aggregate gap (24.3%) without detailing the human evaluation protocol or task breakdown.

## Evidence (verbatim from paper)

> We curated the Sphinx benchmark to include 2,500 questions, with 100 instances per task. We evaluated three proprietary variants of GPT-5 (regular, mini, and nano) using their default reasoning settings*. In addition, we evaluated nine open-source vision-language models, including the Qwen2.5-VL family*, Llama 3.2*, InternVL3*, and the Qwen3-VL family*, covering parameter scales from 3B to 38B. The evaluation prompt is provided in the Appendix[G]. We use the mathruler library* for answer extraction and verification, with GPT-5-mini serving as a fallback when extraction is ambiguous. For open-source models, we use the VLMEvalKit framework for inference*. Results are summarized in Table[1]. Overall, GPT-5 achieves the best performance with an average accuracy of 51.1% on all tasks, although it still falls short of human accuracy by 24.3%.

## Citation

```bibtex
@misc{alam2025sphinx,
  title={SPHINX: A Synthetic Environment for Visual Perception and Reasoning},
  author={Alam et al. (2025)},
  year={2025},
  note={arXiv:2511.20814}
}
```

- arXiv: 2511.20814