Surds Eval

surds-eval

SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models — Guo et al. (2024) (arXiv:2411.13112, 2024)

What this evaluates

This benchmark evaluates fine-grained spatial understanding and reasoning capabilities of vision-language models in real-world driving scenarios. It probes six distinct spatial dimensions: orientation (Yaw), pixel-level localization, depth estimation, pairwise distance, lateral ordering, and front-back relations.

Datasets

• SURDS — total ?; splits: test (-1); repo https://github.com/XiandaGuo/Drive-MLLM

Metrics

• Score (primary) — range: [0, 100]
  • For the Pixel Localization task, a centerness-based metric is used. For the other five tasks (Yaw, Depth, Dis, L/R, F/B), a prediction receives 1 if it exactly matches the ground-truth answer and 0 otherwise. Each task's score is the average over all N QA pairs, scaled to a percentage. The final Score is the unweighted average of the six individual task scores.

Input / output format

Input: A driving scene image paired with a standardized instruction/question and strict output format constraints.

Output: A structured response that must strictly adhere to a predefined format specified by the benchmark.

Scoring recipe

def compute_score(predictions, golds):
    task_scores = {}
    for task in ['Yaw', 'Pixel', 'Depth', 'Dis', 'L/R', 'F/B']:
        if task == 'Pixel':
            task_scores[task] = compute_centerness_metric(predictions[task], golds[task])
        else:
            matches = [1 if p == g else 0 for p, g in zip(predictions[task], golds[task])]
            task_scores[task] = (sum(matches) / len(matches)) * 100
    return sum(task_scores.values()) / len(task_scores)

Common pitfalls

• The Pixel Localization task uses a centerness-based metric rather than exact-match accuracy, unlike the other five tasks.
• The overall Score is a simple unweighted average across six spatial dimensions, which can mask significant performance gaps between single-object and multi-object reasoning tasks.
• Models must strictly follow the predefined output format; failure to do so prevents parsing and evaluation.

Evidence (verbatim from paper)

The Score column represents the average performance across these six metrics. For the Pixel Localization Estimation task, we adopt a centerness-based metric. For other tasks, a prediction receives a score of 1 if it matches the ground-truth answer, and 0 otherwise. Given N QA pairs, the metric score for each task is computed as the average over all N pairs. The final overall score is the average of all individual task scores.

Citation

@misc{guo2024surds,
  title={SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models},
  author={Guo et al. (2024)},
  year={2024},
  note={arXiv:2411.13112}
}

• arXiv: 2411.13112