Spirit LM: Interleaved Spoken and Written Language Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916445249601536 |
|---|---|
| author | Nguyen, Tu Anh Muller, Benjamin Yu, Bokai Costa-jussa, Marta R. Elbayad, Maha Popuri, Sravya Ropers, Christophe Duquenne, Paul-Ambroise Algayres, Robin Mavlyutov, Ruslan Gat, Itai Williamson, Mary Synnaeve, Gabriel Pino, Juan Sagot, Benoit Dupoux, Emmanuel |
| author_facet | Nguyen, Tu Anh Muller, Benjamin Yu, Bokai Costa-jussa, Marta R. Elbayad, Maha Popuri, Sravya Ropers, Christophe Duquenne, Paul-Ambroise Algayres, Robin Mavlyutov, Ruslan Gat, Itai Williamson, Mary Synnaeve, Gabriel Pino, Juan Sagot, Benoit Dupoux, Emmanuel |
| contents | We introduce Spirit LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. Spirit LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that Spirit LM can learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification). We make available model weights and inference code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05755 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Spirit LM: Interleaved Spoken and Written Language Model Nguyen, Tu Anh Muller, Benjamin Yu, Bokai Costa-jussa, Marta R. Elbayad, Maha Popuri, Sravya Ropers, Christophe Duquenne, Paul-Ambroise Algayres, Robin Mavlyutov, Ruslan Gat, Itai Williamson, Mary Synnaeve, Gabriel Pino, Juan Sagot, Benoit Dupoux, Emmanuel Computation and Language Sound Audio and Speech Processing We introduce Spirit LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically-curated speech-text parallel corpus. Spirit LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that Spirit LM can learn new tasks in a few-shot fashion across modalities (i.e. ASR, TTS, Speech Classification). We make available model weights and inference code. |
| title | Spirit LM: Interleaved Spoken and Written Language Model |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2402.05755 |