Embspatial Bench Eval

embspatial-bench-eval

Embspacial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models — Du et al. (2024) (arXiv:2406.05756, 2024)

What this evaluates

Evaluates large vision-language models' ability to reason about egocentric spatial relations (e.g., above, below, left, right, close, far) within 3D embodied environments. It probes whether models can accurately localize objects and identify spatial configurations from a first-person perspective.

Datasets

• Embspacial-Bench — total ?; splits: test (-1); repo https://github.com/mengfeidu/EmbSpatial-Bench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted options out of the total number of queries. Calculated separately for generation-based and likelihood-based evaluation strategies.

Input / output format

Input: Egocentric visual scene (image or 3D representation) paired with a natural language query asking about spatial relations between objects.

Output: For generation strategy: a textual prediction of the correct option. For likelihood strategy: token probabilities for each candidate option.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Generation-based accuracy is heavily penalized by instruction-following failures rather than pure spatial reasoning errors.
• Egocentric spatial relations are viewpoint-dependent; models often confuse left/right or close/far when the camera orientation changes.
• Likelihood strategy requires access to internal token probabilities, which is not available for all closed-source models.

Evidence (verbatim from paper)

Based on EmbSpatial-Bench, we conduct zero-shot evaluation of current LVLMs, using accuracy as the metric. Two evaluation strategies are employed. The first one is the generation-based strategy, which directly uses predicted options from the textual outputs of models. Considering the insufficient instruction-following ability of some LVLMs, we also employed a likelihood strategy, using the option with the highest probability generated by the model

Citation

@misc{du2024embspatialbench,
  title={Embspacial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2406.05756}
}

• arXiv: 2406.05756