semantic-audio-visual-navigation

name	semantic-audio-visual-navigation
title	Semantic Audio-Visual Navigation in Continuous Environments
version	0.0.3
engine	skillxiv-v0.0.3-claude-opus-4.6
license	MIT
url	https://arxiv.org/abs/2603.19660
keywords	["Embodied AI","Audio-Visual Navigation","Multimodal Fusion","Continuous Control","Goal Tracking"]
description	Enable agents to navigate toward sound-emitting objects in continuous 3D spaces with dynamic audio (intermittent sounds, silent periods). Integrate memory-augmented goal descriptors with binaural audio processing and self-motion cues to maintain goal representations even after auditory signals cease.

Semantic Audio-Visual Navigation in Continuous Environments

Problem Statement & Task Definition

SAVN-CE Task: Navigate embodied agents toward intermittently-emitting sound sources in continuous 3D indoor environments with realistic audio rendering.

Key Complexity: Unlike prior discrete-grid navigation, agents move with fine-grained continuous actions (0.25m translations, 15° rotations) and must track goals through silence periods when audio signals vanish.

Realism: Uses real-time binaural audio rendering instead of precomputed impulse responses—agents hear dynamic acoustic cues as they move through space.

Component Innovation: Memory-Augmented Goal Descriptor

The Modification: Extend standard visual navigation to continuous environments by adding memory-augmented goal descriptor network (MAGNet) that fuses audio, self-motion, and episodic memory for robust goal tracking during silence.

Three-Component Architecture:

1. Multimodal Observation Encoder:

   • RGB-D image processing
   • Binaural waveform audio features
   • Agent egomotion (velocity, angular velocity)
   • Previous action history

2. Memory-Augmented Goal Descriptor Network (GDN):

   • Accumulates auditory cues across time steps
   • Integrates self-motion to predict goal location during silence
   • Maintains episodic memory of past audio observations
   • Outputs ACCDDOA representation:
     • Activity: Is goal currently emitting?
     • Direction-of-Arrival (DoA): Angular direction to goal
     • Distance: Estimated distance to goal

# Goal descriptor memory mechanism:
# Maintains goal state g_t combining:
# - Current audio cue a_t (binaural waveform features)
# - Self-motion delta: Δx_t, Δθ_t (translation + rotation)
# - Episodic memory: M = {(a_τ, x_τ) for τ < t}
#
# Memory-updated goal estimate:
# g_t = GDN(a_t, Δx_t, Δθ_t, M)
# → outputs (activity, direction_of_arrival, distance)

3. Context-Aware Policy Network:
   • Transformer-based decoder