Investigating Decoder-only Large Language Models for Speech-to-text Translation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914857562931200 |
|---|---|
| author | Huang, Chao-Wei Lu, Hui Gong, Hongyu Inaguma, Hirofumi Kulikov, Ilia Mavlyutov, Ruslan Popuri, Sravya |
| author_facet | Huang, Chao-Wei Lu, Hui Gong, Hongyu Inaguma, Hirofumi Kulikov, Ilia Mavlyutov, Ruslan Popuri, Sravya |
| contents | Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03169 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating Decoder-only Large Language Models for Speech-to-text Translation Huang, Chao-Wei Lu, Hui Gong, Hongyu Inaguma, Hirofumi Kulikov, Ilia Mavlyutov, Ruslan Popuri, Sravya Computation and Language Sound Audio and Speech Processing Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT. |
| title | Investigating Decoder-only Large Language Models for Speech-to-text Translation |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2407.03169 |