Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.08524 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908403624837120 |
|---|---|
| author | Wang, Weiguo Nie, Andy Zhou, Wenrui Kai, Yi Hu, Chengchen |
| author_facet | Wang, Weiguo Nie, Andy Zhou, Wenrui Kai, Yi Hu, Chengchen |
| contents | Large Language Models (LLMs) have shown remarkable capabilities in text and multimodal processing, yet they fundamentally lack physical awareness--understanding of real-world physical phenomena. In this work, we present ACORN, a framework that teaches LLMs physical awareness through sound, focusing on fundamental physical phenomena like the Doppler effect, multipath effect, and spatial relationships. To overcome data scarcity, ACORN introduce a physics-based simulator combining real-world sound sources with controlled physical channels to generate diverse training data. Using this simulator, we build AQA-PHY, a comprehensive Audio Question-Answer dataset, and propose an audio encoder that processes both magnitude and phase information. By connecting our audio encoder to state-of-the-art LLMs, we demonstrate reasonable results in both simulated and real-world tasks, such as line-of-sight detection, Doppler effect estimation, and Direction-of-Arrival estimation, paving the way for enabling LLMs to understand physical world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08524 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Teaching Physical Awareness to LLMs through Sounds Wang, Weiguo Nie, Andy Zhou, Wenrui Kai, Yi Hu, Chengchen Sound Artificial Intelligence Multimedia Robotics Audio and Speech Processing Large Language Models (LLMs) have shown remarkable capabilities in text and multimodal processing, yet they fundamentally lack physical awareness--understanding of real-world physical phenomena. In this work, we present ACORN, a framework that teaches LLMs physical awareness through sound, focusing on fundamental physical phenomena like the Doppler effect, multipath effect, and spatial relationships. To overcome data scarcity, ACORN introduce a physics-based simulator combining real-world sound sources with controlled physical channels to generate diverse training data. Using this simulator, we build AQA-PHY, a comprehensive Audio Question-Answer dataset, and propose an audio encoder that processes both magnitude and phase information. By connecting our audio encoder to state-of-the-art LLMs, we demonstrate reasonable results in both simulated and real-world tasks, such as line-of-sight detection, Doppler effect estimation, and Direction-of-Arrival estimation, paving the way for enabling LLMs to understand physical world. |
| title | Teaching Physical Awareness to LLMs through Sounds |
| topic | Sound Artificial Intelligence Multimedia Robotics Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.08524 |