Vln Task Planning Eval

vln-task-planning-eval

Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph — Zhao et al. (2024) (arXiv:2408.02535, 2024)

What this evaluates

Evaluates an agent's ability to decompose coarse-grained natural language navigation instructions into executable subtasks and navigate through simulated environments to reach target locations or interact with objects. It probes task planning, visual-language grounding, and dynamic error recovery in continuous or discrete navigation spaces.

Datasets

• R2R — total ?; splits: test (-1)
• REVERIE — total ?; splits: test (-1)
• ALFRED — total ?; splits: test (-1)

Metrics

• Success Rate (SR) (primary) — range: [0, 1]
  • The ratio of episodes where the agent stops within 3 meters of the target location.
• Success Path Length (SPL) — range: [0, 1]
  • The success rate weighted by the normalized inverse of the path length (expert path length divided by agent path length).
• Navigation Error (NE) — range: meters
  • The average distance in meters between the agent's final position and the target.
• Trajectory Length (TL) — range: meters
  • The average path length in meters traversed by the agent.
• Oracle Success Rate (OSR) — range: [0, 1]
  • The ratio of trajectories that pass through at least one viewpoint where the target object is visible.
• Conditional Success (GC) — range: [0, 1]
  • The ratio of subtasks successfully completed over the entire task.
• Path length weighted SR (PLWSR) — range: [0, 1]
  • The ratio of the expert path length to the length taken by the agent.
• Path length weighted GC (PLWGC) — range: [0, 1]
  • The ratio of the ground truth path length to the length taken by the agent.

Input / output format

Input: Coarse-grained natural language instruction and step-wise panoramic visual observations (local views). For ALFRED, interactive steps also require identifying target object masks.

Output: Discrete action selection: choose a navigable viewpoint to move to, or for ALFRED, select an action category (movement/interaction) and output a target object mask.

Scoring recipe

def compute_sr(final_pos, target_pos, threshold=3.0):
    return 1.0 if distance(final_pos, target_pos) <= threshold else 0.0

def compute_spl(final_pos, target_pos, agent_path_len, expert_path_len, threshold=3.0):
    sr = 1.0 if distance(final_pos, target_pos) <= threshold else 0.0
    if expert_path_len == 0: return 0.0
    return sr * (expert_path_len / agent_path_len)

def compute_ne(final_pos, target_pos):
    return distance(final_pos, target_pos)

def compute_tl(agent_path_len):
    return agent_path_len

Common pitfalls

• The evaluation uses coarse-grained instructions without oracle fine-grained subtasks, so planners must handle ambiguous or high-level commands without ground-truth decomposition.
• The dynamic backtracking thresholds (x=0.25, W varies by dataset) are critical; setting them too low causes premature termination, while too high leads to step-limit failures.
• ALFRED requires joint action and object mask prediction, unlike R2R/REVERIE which only require viewpoint selection, complicating direct metric comparison across benchmarks.

Evidence (verbatim from paper)

For R2R: Trajectory Length (TL): the average path length in meters; Navigation Error (NE): the average distance between the agent’s final position and the target in meters; Success Rate (SR): the ratio of stopping within $3$ meters to the target; Success Path Length (SPL): the success rate weighted by the normalized inverse of the Path Length.

Citation

@misc{zhao2024towards,
  title={Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph},
  author={Zhao et al. (2024)},
  year={2024},
  note={arXiv:2408.02535}
}

• arXiv: 2408.02535