Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917224098299904 |
|---|---|
| author | Wang, Hankun Wang, Haoran Guo, Yiwei Li, Zhihan Du, Chenpeng Yu, Kai |
| author_facet | Wang, Hankun Wang, Haoran Guo, Yiwei Li, Zhihan Du, Chenpeng Yu, Kai |
| contents | Although text-based large language models exhibit human-level writing ability and remarkable intelligence, speech language models (SLMs) still struggle to generate semantically coherent outputs. There are several potential reasons for this performance degradation: (A) speech tokens mainly provide phonetic information rather than semantic information, (B) the length of speech sequences is much longer than that of text sequences, and (C) paralinguistic information, such as prosody, introduces additional complexity and variability. In this paper, we explore the influence of three key factors separately by transiting the modality from text to speech in an evolving manner. Our findings reveal that the impact of the three factors varies. Factor A has a relatively minor impact, factor B influences syntactical and semantic modeling more obviously, and factor C exerts the most significant impact, particularly in the basic lexical modeling. Based on these findings, we provide insights into the unique challenges of training SLMs and highlight pathways to develop more effective end-to-end SLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_17048 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective Wang, Hankun Wang, Haoran Guo, Yiwei Li, Zhihan Du, Chenpeng Yu, Kai Audio and Speech Processing Computation and Language Sound Although text-based large language models exhibit human-level writing ability and remarkable intelligence, speech language models (SLMs) still struggle to generate semantically coherent outputs. There are several potential reasons for this performance degradation: (A) speech tokens mainly provide phonetic information rather than semantic information, (B) the length of speech sequences is much longer than that of text sequences, and (C) paralinguistic information, such as prosody, introduces additional complexity and variability. In this paper, we explore the influence of three key factors separately by transiting the modality from text to speech in an evolving manner. Our findings reveal that the impact of the three factors varies. Factor A has a relatively minor impact, factor B influences syntactical and semantic modeling more obviously, and factor C exerts the most significant impact, particularly in the basic lexical modeling. Based on these findings, we provide insights into the unique challenges of training SLMs and highlight pathways to develop more effective end-to-end SLMs. |
| title | Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2412.17048 |