Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916299684184064 |
|---|---|
| author | Trinh, Viet Anh Southwell, Rosy Guan, Yiwen He, Xinlu Wang, Zhiyong Whitehill, Jacob |
| author_facet | Trinh, Viet Anh Southwell, Rosy Guan, Yiwen He, Xinlu Wang, Zhiyong Whitehill, Jacob |
| contents | Recent work on discrete speech tokenization has paved the way for models that can seamlessly perform multiple tasks across modalities, e.g., speech recognition, text to speech, speech to speech translation. Moreover, large language models (LLMs) pretrained from vast text corpora contain rich linguistic information that can improve accuracy in a variety of tasks. In this paper, we present a decoder-only Discrete Multimodal Language Model (DMLM), which can be flexibly applied to multiple tasks (ASR, T2S, S2TT, etc.) and modalities (text, speech, vision). We explore several critical aspects of discrete multi-modal models, including the loss function, weight initialization, mixed training supervision, and codebook. Our results show that DMLM benefits significantly, across multiple tasks and datasets, from a combination of supervised and unsupervised training. Moreover, for ASR, it benefits from initializing DMLM from a pretrained LLM, and from a codebook derived from Whisper activations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06582 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing Trinh, Viet Anh Southwell, Rosy Guan, Yiwen He, Xinlu Wang, Zhiyong Whitehill, Jacob Computation and Language Machine Learning Audio and Speech Processing Recent work on discrete speech tokenization has paved the way for models that can seamlessly perform multiple tasks across modalities, e.g., speech recognition, text to speech, speech to speech translation. Moreover, large language models (LLMs) pretrained from vast text corpora contain rich linguistic information that can improve accuracy in a variety of tasks. In this paper, we present a decoder-only Discrete Multimodal Language Model (DMLM), which can be flexibly applied to multiple tasks (ASR, T2S, S2TT, etc.) and modalities (text, speech, vision). We explore several critical aspects of discrete multi-modal models, including the loss function, weight initialization, mixed training supervision, and codebook. Our results show that DMLM benefits significantly, across multiple tasks and datasets, from a combination of supervised and unsupervised training. Moreover, for ASR, it benefits from initializing DMLM from a pretrained LLM, and from a codebook derived from Whisper activations. |
| title | Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing |
| topic | Computation and Language Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.06582 |