JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916060239757312 |
|---|---|
| author | Liu, Zhan Tang, Changli Wang, Yuxin Zhu, Zhiyuan Chen, Youjun Shao, Yiwen Wang, Tianzi Ke, Lei Jin, Zengrui Zhang, Chao |
| author_facet | Liu, Zhan Tang, Changli Wang, Yuxin Zhu, Zhiyuan Chen, Youjun Shao, Yiwen Wang, Tianzi Ke, Lei Jin, Zengrui Zhang, Chao |
| contents | Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and datasets are available at https://github.com/liuzhan22/JAEGER. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18527 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments Liu, Zhan Tang, Changli Wang, Yuxin Zhu, Zhiyuan Chen, Youjun Shao, Yiwen Wang, Tianzi Ke, Lei Jin, Zengrui Zhang, Chao Computer Vision and Pattern Recognition Artificial Intelligence Sound Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundamental dimensionality mismatch that precludes reliable source localization and spatial reasoning in complex 3D environments. We address this limitation by presenting JAEGER, a framework that extends AV-LLMs to 3D space, to enable joint spatial grounding and reasoning through the integration of RGB-D observations and multi-channel first-order ambisonics. A core contribution of our work is the neural intensity vector (Neural IV), a learned spatial audio representation that encodes robust directional cues to enhance direction-of-arrival estimation, even in adverse acoustic scenarios with overlapping sources. To facilitate large-scale training and systematic evaluation, we propose SpatialSceneQA, a benchmark of 61k instruction-tuning samples curated from simulated physical environments. Extensive experiments demonstrate that our approach consistently surpasses 2D-centric baselines across diverse spatial perception and reasoning tasks, underscoring the necessity of explicit 3D modelling for advancing AI in physical environments. Our source code, pre-trained model checkpoints, and datasets are available at https://github.com/liuzhan22/JAEGER. |
| title | JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Sound |
| url | https://arxiv.org/abs/2602.18527 |