Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.17097 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917281627373568 |
|---|---|
| author | Chen, William Seetharaman, Prem Kumar, Rithesh Nieto, Oriol Watanabe, Shinji Salamon, Justin Jin, Zeyu |
| author_facet | Chen, William Seetharaman, Prem Kumar, Rithesh Nieto, Oriol Watanabe, Shinji Salamon, Justin Jin, Zeyu |
| contents | Despite recent breakthroughs, audio foundation models struggle in processing complex multi-source acoustic scenes. We refer to this challenging domain as audio stories, which can have multiple speakers and background/foreground sound effects. Compared to traditional audio processing tasks, audio stories introduce new layers of semantic, temporal, and physical complexity. To address this challenge, we propose AudioChat, a framework for developing audio foundation models that can generate, edit, and understand audio stories. AudioChat introduces a new paradigm in which LLM-based toolcalling agents simulate interactions between users and the system, and these simulated dialogues are used as training data. We also introduce a novel Audio Transfusion Forcing objective to train the AudioChat model, allowing it to simultaneously decompose high-level instructions via structured chain-of-thought reasoning and perform interactive multi-turn audio understanding/generation. To evaluate generation and editing performance, we develop three new metrics that directly measure task performance instead of relying upon distribution-based scoring. We highly encourage readers to visit our demo to better understand the capabilities of AudioChat: https://wanchichen.github.io/audiochat/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_17097 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing Chen, William Seetharaman, Prem Kumar, Rithesh Nieto, Oriol Watanabe, Shinji Salamon, Justin Jin, Zeyu Sound Despite recent breakthroughs, audio foundation models struggle in processing complex multi-source acoustic scenes. We refer to this challenging domain as audio stories, which can have multiple speakers and background/foreground sound effects. Compared to traditional audio processing tasks, audio stories introduce new layers of semantic, temporal, and physical complexity. To address this challenge, we propose AudioChat, a framework for developing audio foundation models that can generate, edit, and understand audio stories. AudioChat introduces a new paradigm in which LLM-based toolcalling agents simulate interactions between users and the system, and these simulated dialogues are used as training data. We also introduce a novel Audio Transfusion Forcing objective to train the AudioChat model, allowing it to simultaneously decompose high-level instructions via structured chain-of-thought reasoning and perform interactive multi-turn audio understanding/generation. To evaluate generation and editing performance, we develop three new metrics that directly measure task performance instead of relying upon distribution-based scoring. We highly encourage readers to visit our demo to better understand the capabilities of AudioChat: https://wanchichen.github.io/audiochat/. |
| title | AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing |
| topic | Sound |
| url | https://arxiv.org/abs/2602.17097 |