Sage Bench Eval

sage-bench-eval

Towards Physically Executable 3D Gaussian for Embodied Navigation — Miao et al. (2025) (arXiv:2510.21307, 2025)

What this evaluates

Evaluates embodied vision-and-language navigation (VLN) capabilities within physically executable 3D Gaussian Splatting environments. It probes a model's ability to follow natural language instructions (high- and low-level), navigate to goals without collisions, and exhibit smooth, natural motion continuity rather than mechanical or wall-hugging behaviors.

Datasets

• SAGE-Bench — total 2000000; splits: train (500000), test (-1)
• VLN-CE (R2R Val-Unseen) — total ?; splits: val-unseen (-1)

Metrics

• SR (primary) — range: [0, 1]
  • Success Rate: binary metric indicating whether the agent reaches within a predefined threshold distance of the goal location at the end of the episode.
• OSR — range: [0, 1]
  • Oracle Success Rate: binary metric indicating whether the agent visits within the threshold distance of the goal at any point during the episode.
• SPL — range: [0, 1]
  • Success weighted by Path Length: SR multiplied by the ratio of the shortest possible path length to the actual path length taken by the agent.
• CR — range: other
  • Collision Rate: frequency or count of collisions with environment geometry during navigation.
• CSR — range: [0, 1]
  • Continuity Success Rate: novel metric that measures inclusive success without requiring exact ground-truth trajectory fitting (formulas detailed in Sec. 3.3).
• ICP — range: [0, 1]
  • Inter-Collision Points: measures sustained collisions during navigation, capturing micro-collisions or wall-hugging that standard CR misses (formulas detailed in Sec. 3.3).
• PS — range: [0, 1]
  • Path Smoothness: evaluates motion continuity by penalizing large, mechanical turning angles in favor of smooth, natural motion (formulas detailed in Sec. 3.3).
• Episode Time — range: seconds
  • Duration in seconds until the episode terminates (goal reached, collision occurs, or max time of 120s is hit).
• Explored Areas — range: other
  • Total area covered by the agent during the episode, used for the No-goalNav task.

Input / output format

Input: RGB-D or 3DGS scene observations paired with natural language navigation instructions (categorized as high-level or low-level).

Output: Sequential navigation actions/trajectories (e.g., move forward, turn left/right) until the goal is reached or the episode terminates.

Scoring recipe

def compute_vln_metrics(traj, goal, shortest_path, threshold=0.5):
    sr = 1.0 if distance(traj[-1], goal) < threshold else 0.0
    osr = 1.0 if any(distance(p, goal) < threshold for p in traj) else 0.0
    spl = sr * (shortest_path / max(len(traj), 1))
    cr = collision_count(traj) / len(traj)
    # CSR, ICP, PS computed per Sec 3.3 protocol
    return {'SR': sr, 'OSR': osr, 'SPL': spl, 'CR': cr}

Common pitfalls

• Conventional metrics like SR and CR fail to capture unnatural navigation behaviors (e.g., prolonged wall-hugging or sustained micro-collisions); CSR, ICP, and PS are required to detect these.
• 3DGS-based scenes require ~33% more training iterations (160 vs 120) to reach the same success rate compared to scanned mesh data, despite offering faster per-frame rendering.
• Models perform significantly worse on high-level instructions compared to low-level step-by-step instructions, revealing a gap in natural language grounding and planning.

Evidence (verbatim from paper)

In addition to the three novel metrics we proposed in Section[3.3] for evaluating the natural continuity of model navigation — CSR, ICP, and PS — we also adopt common metrics used in VLN tasks, including success rate (SR), oracle success rate (OSR), and success weighted by path length (SPL) and Collision Rate (CR).

Citation

@misc{miao2025sage3d,
  title={Towards Physically Executable 3D Gaussian for Embodied Navigation},
  author={Miao et al. (2025)},
  year={2025},
  note={arXiv:2510.21307}
}

• arXiv: 2510.21307