Socialnav Sub Eval

socialnav-sub-eval

SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation — Munje et al. (2025) (arXiv:2509.08757, 2025)

What this evaluates

Evaluates Vision-Language Models' ability to perform spatial, spatiotemporal, and social reasoning in dynamic, crowd-filled robot navigation scenarios. It probes scene understanding by asking VLMs to answer visual questions based on image sequences and Bird's Eye View (BEV) representations.

Datasets

• SocialNav-SUB — total ?; splits: test (-1)

Metrics

• PA (primary) — range: [0, 1]
  • Probability of Agreement: measures the proportion of VLM answers that exactly match the human dataset responses for each question.
• CWPA — range: [0, 1]
  • Consensus Weighted Probability of Agreement: weights the agreement scores by the level of consensus among human annotators for each question, rewarding correct answers on questions with high human agreement.

Input / output format

Input: Visual question prompts paired with image sequences and Bird's Eye View (BEV) scene representations.

Output: Textual answers to the VQA prompts.

Scoring recipe

def compute_pa_cwpa(predictions, gold_answers, human_consensus):
    agreements = [1.0 if p == g else 0.0 for p, g in zip(predictions, gold_answers)]
    pa = sum(agreements) / len(agreements)
    cwpa = sum(a * c for a, c in zip(agreements, human_consensus)) / sum(human_consensus)
    return pa, cwpa

Common pitfalls

• Assuming state-of-the-art VLMs outperform simple rule-based baselines on this task, when the rule-based system actually achieves higher PA/CWPA.
• Overlooking that performance heavily degrades in high crowd-density scenarios, even for top models.
• Ignoring the significant performance boost from Chain-of-Thought (CoT) prompting and BEV representations, which are critical for fair comparison.

Evidence (verbatim from paper)

The responses generated by the VLM are then compared against human responses from the human dataset using the PA and CWPA metrics, previously defined in Equations [1] and [2] ... Table 1: Average Performance Across Question Categories. The metrics used are PA and CWPA for all questions and for each question category, along with standard error across the questions.

Citation

@misc{munje2025socialnavsub,
  title={SocialNav-SUB: Benchmarking VLMs for Scene Understanding in Social Robot Navigation},
  author={Munje et al. (2025)},
  year={2025},
  note={arXiv:2509.08757}
}

• arXiv: 2509.08757