Dynamic Audio Visual Nav Eval

dynamic-audio-visual-nav-eval

Dynamical Audio-Visual Navigation: Catching Unheard Moving Sound Sources in Unmapped 3D Environments — Younes (2022) (arXiv:2201.04279, 2022)

What this evaluates

Evaluates an embodied agent's ability to navigate towards and catch a moving, previously unheard sound source in unmapped 3D environments using only audio and visual observations. It probes spatial reasoning, temporal memory, and robustness to noisy or distractor audio scenarios.

Datasets

• Replica — total ?; splits: (unstated)
• Matterport3D — total ?; splits: (unstated)

Metrics

• DSPL (primary) — range: [0, 1]
  • Dynamic Success weighted by Path Length. Calculated as the average over N episodes of S_i * (g_i / max(p_i, g_i)), where S_i is 1 if the episode is successful, g_i is the shortest geodesic distance from the agent's start to the earliest catchable position of the moving target, and p_i is the actual path length taken by the agent.

Input / output format

Input: Per step: depth image d_t and binaural spectrogram b_t (left and right ear channels). No displacement vector or goal coordinates provided.

Output: Discrete navigation actions: Move Forward, Rotate Left, Rotate Right, or Stop. (Policy outputs close-by waypoints executed by a Dijkstra planner).

Scoring recipe

def compute_dspl(episodes):
    total = 0.0
    for ep in episodes:
        success = 1.0 if ep.agent_executed_stop_at_target else 0.0
        g_i = geodesic_distance(start_pos, earliest_catchable_pos)
        p_i = agent_path_length
        total += success * (g_i / max(p_i, g_i))
    return total / len(episodes)

Common pitfalls

• DSPL uses an oracle upper bound for g_i based on the earliest position the target could have been caught, which assumes knowledge of the target's trajectory and is not achievable in practice without prior information.
• Standard SPL cannot be used because the goal location changes continuously during the episode, requiring dynamic recalculation of the shortest path to the moving target.

Evidence (verbatim from paper)

The Success weighted by Path Length (SPL) [69] serves as the primary metric to evaluate the navigation performance of embodied agents. However, this metric relies on the prior knowledge of the shortest path to the goal location, which is not available in the newly introduced dynamic AudioGoal task as the goal location changes continually during the episode. Consequently, we introduce the Dynamic Success weighted by Path Length (DSPL) metric, which considers the shortest path length as the geodesic distance between the agent's starting position and the earliest position the moving source passed by if the agent had enough steps from the beginning of the episode to catch the source in that position. The DSPL is defined as follows: DSPL = 1/N Σ S_i * (g_i / max(p_i, g_i)).

Citation

@misc{younes2022dynamical,
  title={Dynamical Audio-Visual Navigation: Catching Unheard Moving Sound Sources in Unmapped 3D Environments},
  author={Younes (2022)},
  year={2022},
  note={arXiv:2201.04279}
}

• arXiv: 2201.04279