adu2021 Semantic Audio-Visual Navigation in Continuous Environments
Problem Statement & Task Definition
SAVN-CE Task: Navigate embodied agents toward intermittently-emitting sound sources in continuous 3D indoor environments with realistic audio rendering.
Key Complexity: Unlike prior discrete-grid navigation, agents move with fine-grained continuous actions (0.25m translations, 15° rotations) and must track goals through silence periods when audio signals vanish.
Realism: Uses real-time binaural audio rendering instead of precomputed impulse responses—agents hear dynamic acoustic cues as they move through space.
Component Innovation: Memory-Augmented Goal Descriptor
The Modification: Extend standard visual navigation to continuous environments by adding memory-augmented goal descriptor network (MAGNet) that fuses audio, self-motion, and episodic memory for robust goal tracking during silence.
Three-Component Architecture:
Multimodal Observation Encoder:
- RGB-D image processing
- Binaural waveform audio features
- Agent egomotion (velocity, angular velocity)
- Previous action history
Memory-Augmented Goal Descriptor Network (GDN):
- Accumulates auditory cues across time steps
- Integrates self-motion to predict goal location during silence
- Maintains episodic memory of past audio observations
- Outputs ACCDDOA representation:
- Activity: Is goal currently emitting?
- Direction-of-Arrival (DoA): Angular direction to goal
- Distance: Estimated distance to goal
# Goal descriptor memory mechanism:
# Maintains goal state g_t combining:
# - Current audio cue a_t (binaural waveform features)
# - Self-motion delta: Δx_t, Δθ_t (translation + rotation)
# - Episodic memory: M = {(a_τ, x_τ) for τ < t}
#
# Memory-updated goal estimate:
# g_t = GDN(a_t, Δx_t, Δθ_t, M)
# → outputs (activity, direction_of_arrival, distance)
- Context-Aware Policy Network:
- Transformer-based decoder
- Conditions on goal descriptor
- Generates continuous navigation actions
What Was Modified for Continuous Environments
Discretization Removal: Prior work used 0.5m grid steps; SAVN-CE uses 0.25m continuous movement with smooth rotation (15° granularity).
Audio Representation: Real-time binaural audio instead of pre-rendered impulse responses—agents receive dynamic acoustic updates reflecting position changes.
Goal Tracking Strategy: Active memory accumulation—system maintains running estimate of goal location even when silent, updated via:
- Audio cues when available (strong signal)
- Self-motion integration when silent (dead-reckoning)
- Episodic memory fusion (comparing current cues to past observations)
Experimental Results
Success Rates (Clean Environments):
- SAVN-CE with MAGNet: 37.7% (baseline from prior discrete methods: 25.6%)
- 12.1% absolute improvement in success rate
- Superior robustness to short-duration sounds and long-distance goals
Key Scenarios:
- Continuous Audio: 48.2% success (best-case)
- **Intermittent Audio (5s on, 5s silent): 37.7% success
- **Very Short Sounds (1s bursts): 22.4% success
Challenging Condition (Mixed Sounds): Performance degrades to 18.3% with competing distractor sounds, indicating room for improved distraction handling.
Dataset & Evaluation
- Environment: Matterport3D with 500K training episodes
- Semantic Categories: 21 object types (chair, table, doorway, kitchen, etc.)
- Distractor Sounds: 102 periodic distractor sounds mixed with goal audio
- Evaluation Metric: Success rate = agent reaches goal within 0.5m and orients correctly
Deployment Considerations
Audio Rendering Fidelity: Real-time binaural rendering requires reasonable spatial audio simulation; pre-rendered IRs may suffice in practice but lose dynamic updates.
Continuous Action Mapping: Map model outputs to robot actuators; 0.25m movements appropriate for humanoid robots, smaller increments for wheeled platforms.
Memory Capacity: MAGNet maintains episodic memory; limit window to last 20-30 observations to avoid unbounded memory growth.
Distractor Robustness: Current system struggles with competing sounds; mitigate by:
- Semantic filtering (goal category labels reduce ambiguity)
- Temporal coherence (favor consistent directions over jittering)
- Multi-hypothesis tracking (maintain multiple goal candidates)
Silence Duration Tolerance: System maintains goal estimates for ~30-60 seconds of silence; beyond that, dead-reckoning diverges and performance collapses.
Practical Implications
- Bridges Simulation-to-Real Gap: Continuous control and realistic audio move beyond toy gridworld navigation
- Multimodal Fusion Value: Memory-augmented architecture shows 12% improvement—memory integration critical for dynamic audio
- Scalability: Tested on indoor environments; outdoor navigation (GPS-less) feasible but untested