Instruction Data Generation and Unsupervised Adaptation for Speech Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917699352788992 |
|---|---|
| author | Noroozi, Vahid Chen, Zhehuai Majumdar, Somshubra Huang, Steve Balam, Jagadeesh Ginsburg, Boris |
| author_facet | Noroozi, Vahid Chen, Zhehuai Majumdar, Somshubra Huang, Steve Balam, Jagadeesh Ginsburg, Boris |
| contents | In this paper, we propose three methods for generating synthetic samples to train and evaluate multimodal large language models capable of processing both text and speech inputs. Addressing the scarcity of samples containing both modalities, synthetic data generation emerges as a crucial strategy to enhance the performance of such systems and facilitate the modeling of cross-modal relationships between the speech and text domains. Our process employs large language models to generate textual components and text-to-speech systems to generate speech components. The proposed methods offer a practical and effective means to expand the training dataset for these models. Experimental results show progress in achieving an integrated understanding of text and speech. We also highlight the potential of using unlabeled speech data to generate synthetic samples comparable in quality to those with available transcriptions, enabling the expansion of these models to more languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12946 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Instruction Data Generation and Unsupervised Adaptation for Speech Language Models Noroozi, Vahid Chen, Zhehuai Majumdar, Somshubra Huang, Steve Balam, Jagadeesh Ginsburg, Boris Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning In this paper, we propose three methods for generating synthetic samples to train and evaluate multimodal large language models capable of processing both text and speech inputs. Addressing the scarcity of samples containing both modalities, synthetic data generation emerges as a crucial strategy to enhance the performance of such systems and facilitate the modeling of cross-modal relationships between the speech and text domains. Our process employs large language models to generate textual components and text-to-speech systems to generate speech components. The proposed methods offer a practical and effective means to expand the training dataset for these models. Experimental results show progress in achieving an integrated understanding of text and speech. We also highlight the potential of using unlabeled speech data to generate synthetic samples comparable in quality to those with available transcriptions, enabling the expansion of these models to more languages. |
| title | Instruction Data Generation and Unsupervised Adaptation for Speech Language Models |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.12946 |