An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916124815261696 |
|---|---|
| author | Ma, Ziyang Yang, Guanrou Yang, Yifan Gao, Zhifu Wang, Jiaming Du, Zhihao Yu, Fan Chen, Qian Zheng, Siqi Zhang, Shiliang Chen, Xie |
| author_facet | Ma, Ziyang Yang, Guanrou Yang, Yifan Gao, Zhifu Wang, Jiaming Du, Zhihao Yu, Fan Chen, Qian Zheng, Siqi Zhang, Shiliang Chen, Xie |
| contents | In this paper, we focus on solving one of the most important tasks in the field of speech processing, i.e., automatic speech recognition (ASR), with speech foundation encoders and large language models (LLM). Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM. We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task. To be more specific, we benchmark and explore various combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup and little task-specific design, where only the linear projector is trained. To the best of our knowledge, SLAM-ASR achieves the best performance on the Librispeech benchmark among LLM-based ASR models and even outperforms the latest LLM-based audio-universal model trained on massive pair data. Finally, we explore the capability emergence of LLM-based ASR in the process of modal alignment. We hope that our study can facilitate the research on extending LLM with cross-modality capacity and shed light on the LLM-based ASR community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_08846 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Embarrassingly Simple Approach for LLM with Strong ASR Capacity Ma, Ziyang Yang, Guanrou Yang, Yifan Gao, Zhifu Wang, Jiaming Du, Zhihao Yu, Fan Chen, Qian Zheng, Siqi Zhang, Shiliang Chen, Xie Computation and Language Artificial Intelligence Multimedia Sound Audio and Speech Processing In this paper, we focus on solving one of the most important tasks in the field of speech processing, i.e., automatic speech recognition (ASR), with speech foundation encoders and large language models (LLM). Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM. We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task. To be more specific, we benchmark and explore various combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup and little task-specific design, where only the linear projector is trained. To the best of our knowledge, SLAM-ASR achieves the best performance on the Librispeech benchmark among LLM-based ASR models and even outperforms the latest LLM-based audio-universal model trained on massive pair data. Finally, we explore the capability emergence of LLM-based ASR in the process of modal alignment. We hope that our study can facilitate the research on extending LLM with cross-modality capacity and shed light on the LLM-based ASR community. |
| title | An Embarrassingly Simple Approach for LLM with Strong ASR Capacity |
| topic | Computation and Language Artificial Intelligence Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2402.08846 |