E3vs Bench Eval

e3vs-bench-eval

E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes — Sakamoto et al. (2026) (arXiv:2604.17969, 2026)

What this evaluates

Probes 5-DoF viewpoint control and active perception in photorealistic 3D scenes. Tests whether vision-language models can navigate, resolve occlusions, and answer questions by strategically selecting viewpoints to gather spatially dependent visual evidence.

Datasets

• E3VS-Bench — total 2014; splits: train (1406), val (231), test (377)

Metrics

• VLM Judge Score (primary) — range: [1, 5]
  • Score of 1 or 5 assigned by a GPT-5.1 judge comparing the predicted response to the ground-truth answer and end/goal images. Validated with Spearman ρ=0.54 against human evaluation.
• Average Steps — range: steps
  • Mean number of navigation steps taken per episode across the test set.
• Collision Rate — range: [0, 1]
  • Fraction of episodes where at least one collision occurs. Calculated as binary indicator (1 if collision, 0 otherwise) averaged over all episodes.

Input / output format

Input: Current viewpoint image(s) at 512×512 resolution with 90° FOV, task question, and optionally a history of previous frames (1, 3, or 5) for memory ablation studies.

Output: Action command (move_forward 0.25m or rotate 30°) or final answer. Max token length is 128 without reasoning or 256+ with reasoning. Defaults to move_forward if no action command is generated within the token limit.

Scoring recipe

# VLM Judge Score
judge = GPT51Judge()
score = judge.evaluate(prediction, ground_truth, end_image, goal_image) # returns 1 or 5

# Average Steps
avg_steps = sum(len(trajectory) for trajectory in trajectories) / len(trajectories)

# Collision Rate
collision_rate = sum(1 if episode_has_collision(ep) else 0 for ep in episodes) / len(episodes)

Common pitfalls

• GPT-5.1 was used for dataset filtering, so it is excluded from blind and start-view baselines to prevent evaluation bias.
• Models may exploit dataset bias by answering from the initial viewpoint without exploring, particularly for binary tasks like OST.
• The max 25-step limit and default move_forward fallback can artificially inflate collision rates or truncate valid exploration trajectories.

Evidence (verbatim from paper)

Regarding answer correctness, we employ a VLM-as-a-judge framework in accordance with OpenEQA, using GPT 5.1 as the evaluator. The judge VLM receives the predicted response and ground-truth answer, along with the end and goal images, and outputs a score of 5 for correct predictions and 1 for incorrect ones. Furthermore, exploration efficiency is quantified by the average number of steps, while navigation safety is measured using Collision Rate, defined as a binary indicator that takes the value 1 if a collision occurs at least once within an episode and 0 otherwise.

Citation

@misc{sakamoto2026e3vsbench,
  title={E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes},
  author={Sakamoto et al. (2026)},
  year={2026},
  note={arXiv:2604.17969}
}

• arXiv: 2604.17969