Ssrbench Eval

ssrbench-eval

SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning — Yang Liu et al. (2025) (arXiv:2505.12448, 2025)

What this evaluates

Evaluates vision-language models on spatial understanding and general question answering using image-text pairs. It probes capabilities such as object existence, attribute recognition, action identification, counting, positional reasoning, and object identification, specifically testing how well models leverage depth information and spatial reasoning.

Datasets

• SSRBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Evaluated using an LLM-assistant (Qwen2.5-14B-Instruct-1M) that compares model-generated answers against ground-truth answers to determine correctness.

Input / output format

Input: Image, corresponding depth map (estimated via Depth Pro), and a natural language question.

Output: Natural language answer string.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        prompt = f'Question: {gold["question"]}\nModel Answer: {pred}\nGround Truth: {gold["answer"]}\nIs the model answer correct? Answer Yes or No.'
        judge_response = llm_assistant.generate(prompt)
        if 'yes' in judge_response.lower():
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Uses an LLM-as-a-judge (Qwen2.5-14B) for scoring instead of exact-match or regex-based evaluation, which can introduce scorer bias or inconsistency.
• Depth maps are not provided as ground truth but are estimated externally using Depth Pro, meaning evaluation performance depends on the depth estimator's accuracy.
• Task-specific metrics (e.g., Position, Existence, Counting) are reported in tables, but the main headline results focus on average accuracy across all tasks.

Evidence (verbatim from paper)

Performance metrics include accuracy as well as a quantitative score ranging from 0 to 5, both are produced using the LLM-Assistant powered by the Qwen2.5-14B-Instruct-1M [[38], [93]]. SSRBench consists of two primary categories, general understanding and spatial understanding, allowing simultaneous evaluation of VLM performance in both general question answering and spatial reasoning tasks.

Citation

@misc{liu2025ssr,
  title={SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning},
  author={Yang Liu et al. (2025)},
  year={2025},
  note={arXiv:2505.12448}
}

• arXiv: 2505.12448