Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.12000 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912645859246080 |
|---|---|
| author | Tian, Jinchuan Lee, Sang-gil Kong, Zhifeng Ghosh, Sreyan Goel, Arushi Yang, Chao-Han Huck Dai, Wenliang Liu, Zihan Ye, Hanrong Watanabe, Shinji Shoeybi, Mohammad Catanzaro, Bryan Valle, Rafael Ping, Wei |
| author_facet | Tian, Jinchuan Lee, Sang-gil Kong, Zhifeng Ghosh, Sreyan Goel, Arushi Yang, Chao-Han Huck Dai, Wenliang Liu, Zihan Ye, Hanrong Watanabe, Shinji Shoeybi, Mohammad Catanzaro, Bryan Valle, Rafael Ping, Wei |
| contents | Recent advances in the audio language modeling (ALM) domain tackle audio understanding and text-to-audio generation as separate tasks. Very few studies attempt to unify these tasks -- an essential step toward advanced multimodal reasoning. This paper introduces U}nified Audio Language Model (UALM), which aims to unify audio understanding, text-to-audio generation, and multimodal reasoning in a single model. To achieve this goal, we first present UALM-Gen, a text-to-audio language model that directly predicts audio tokens and is comparable to state-of-the-art diffusion-based models. We then demonstrate, using proper data blending, training recipes, and inference techniques, that our single UALM model matches the quality of state-of-the-art specialized models in audio understanding, text-to-audio generation, and text reasoning. Furthermore, we present UALM-Reason, a multimodal reasoning model that utilizes both text and audio in the intermediate thinking steps to facilitate complex generation tasks. To our knowledge, this is the first demonstration in audio research of cross-modal generative reasoning, with its effectiveness confirmed by subjective evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_12000 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UALM: Unified Audio Language Model for Understanding, Generation and Reasoning Tian, Jinchuan Lee, Sang-gil Kong, Zhifeng Ghosh, Sreyan Goel, Arushi Yang, Chao-Han Huck Dai, Wenliang Liu, Zihan Ye, Hanrong Watanabe, Shinji Shoeybi, Mohammad Catanzaro, Bryan Valle, Rafael Ping, Wei Sound Computation and Language Machine Learning Recent advances in the audio language modeling (ALM) domain tackle audio understanding and text-to-audio generation as separate tasks. Very few studies attempt to unify these tasks -- an essential step toward advanced multimodal reasoning. This paper introduces U}nified Audio Language Model (UALM), which aims to unify audio understanding, text-to-audio generation, and multimodal reasoning in a single model. To achieve this goal, we first present UALM-Gen, a text-to-audio language model that directly predicts audio tokens and is comparable to state-of-the-art diffusion-based models. We then demonstrate, using proper data blending, training recipes, and inference techniques, that our single UALM model matches the quality of state-of-the-art specialized models in audio understanding, text-to-audio generation, and text reasoning. Furthermore, we present UALM-Reason, a multimodal reasoning model that utilizes both text and audio in the intermediate thinking steps to facilitate complex generation tasks. To our knowledge, this is the first demonstration in audio research of cross-modal generative reasoning, with its effectiveness confirmed by subjective evaluations. |
| title | UALM: Unified Audio Language Model for Understanding, Generation and Reasoning |
| topic | Sound Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.12000 |