Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Grounding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912536218042368 |
|---|---|
| author | Cao-Dinh, Duc Le-Duc, Khai Dao, Anh Tat, Bach Phan Ngo, Chris Nguyen, Duy M. H. Khanh, Nguyen X. Nguyen-Tang, Thanh |
| author_facet | Cao-Dinh, Duc Le-Duc, Khai Dao, Anh Tat, Bach Phan Ngo, Chris Nguyen, Duy M. H. Khanh, Nguyen X. Nguyen-Tang, Thanh |
| contents | 3D Visual Grounding (3DVG) involves localizing target objects in 3D point clouds based on natural language. While prior work has made strides using textual descriptions, leveraging spoken language-known as Audio-based 3D Visual Grounding-remains underexplored and challenging. Motivated by advances in automatic speech recognition (ASR) and speech representation learning, we propose Audio-3DVG, a simple yet effective framework that integrates audio and spatial information for enhanced grounding. Rather than treating speech as a monolithic input, we decompose the task into two complementary components. First, we introduce (i) Object Mention Detection, a multi-label classification task that explicitly identifies which objects are referred to in the audio, enabling more structured audio-scene reasoning. Second, we propose an (ii) Audio-Guided Attention module that models the interactions between target candidates and mentioned objects, enhancing discrimination in cluttered 3D environments. To support benchmarking, we (iii) synthesize audio descriptions for standard 3DVG datasets, including ScanRefer, Sr3D, and Nr3D. Experimental results demonstrate that Audio-3DVG not only achieves new state-of-the-art performance in audio-based grounding, but also competes with text-based methods, highlight the promise of integrating spoken language into 3D vision tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00669 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Grounding Cao-Dinh, Duc Le-Duc, Khai Dao, Anh Tat, Bach Phan Ngo, Chris Nguyen, Duy M. H. Khanh, Nguyen X. Nguyen-Tang, Thanh Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Robotics 3D Visual Grounding (3DVG) involves localizing target objects in 3D point clouds based on natural language. While prior work has made strides using textual descriptions, leveraging spoken language-known as Audio-based 3D Visual Grounding-remains underexplored and challenging. Motivated by advances in automatic speech recognition (ASR) and speech representation learning, we propose Audio-3DVG, a simple yet effective framework that integrates audio and spatial information for enhanced grounding. Rather than treating speech as a monolithic input, we decompose the task into two complementary components. First, we introduce (i) Object Mention Detection, a multi-label classification task that explicitly identifies which objects are referred to in the audio, enabling more structured audio-scene reasoning. Second, we propose an (ii) Audio-Guided Attention module that models the interactions between target candidates and mentioned objects, enhancing discrimination in cluttered 3D environments. To support benchmarking, we (iii) synthesize audio descriptions for standard 3DVG datasets, including ScanRefer, Sr3D, and Nr3D. Experimental results demonstrate that Audio-3DVG not only achieves new state-of-the-art performance in audio-based grounding, but also competes with text-based methods, highlight the promise of integrating spoken language into 3D vision tasks. |
| title | Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Grounding |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2507.00669 |