Ssi Bench Eval

ssi-bench-eval

Thinking in Structures: Evaluating Spatial Intelligence through Reasoning on Constrained Manifolds — Chen Yang et al. (2026) (arXiv:2602.07864, 2026)

What this evaluates

Probes constrained-manifold spatial reasoning by requiring models to rank structural components based on geometric, topological, and physical constraints in complex 3D engineering scenes. It tests compositional spatial operations like mental rotation, occlusion handling, and force-path reasoning, revealing gaps in structural grounding and 3D constraint consistency.

Datasets

• SSI-Bench — total 1000; splits: test (1000)

Metrics

• Taskwise Accuracy (primary) — range: [0, 1]
  • Exact-match accuracy on the full permutation of ranked indices.
• Pairwise Accuracy — range: [0, 1]
  • Fraction of correctly ordered pairs relative to the ground-truth ranking.

Input / output format

Input: Input images (resized so the longer side is at most 512 pixels) paired with a unified prompt template specific to each question type.

Output: A parsable Python list encoding a permutation of indices representing the ranked order of structural components.

Scoring recipe

def compute_metrics(pred, gold):
    taskwise = 1.0 if pred == gold else 0.0
    n = len(pred)
    correct_pairs = sum(1 for i in range(n) for j in range(i+1, n)
                        if (pred[i] < pred[j]) == (gold[i] < gold[j]))
    pairwise = correct_pairs / (n * (n - 1) / 2)
    return taskwise, pairwise

Common pitfalls

• Models often exploit 2D shortcut cues rather than performing true 3D structural reasoning, leading to poor generalization on constrained manifolds.
• High 'thinking' token usage does not guarantee correct reasoning and can amplify errors when models deliberate over incorrect structural hypotheses.
• The random baseline for ranking is 12.85%, so models scoring near this level are effectively guessing rather than reasoning.

Evidence (verbatim from paper)

We report two complementary metrics: Taskwise Accuracy (exact-match accuracy on the full permutation) and Pairwise Accuracy (pairwise ordering consistency).

Citation

@misc{yang2026thinking,
  title={Thinking in Structures: Evaluating Spatial Intelligence through Reasoning on Constrained Manifolds},
  author={Chen Yang et al. (2026)},
  year={2026},
  note={arXiv:2602.07864}
}

• arXiv: 2602.07864