Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.17353 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912101911494656 |
|---|---|
| author | Yuen, Robin Shing-Hei Tse, Timothy Tin-Long Zhu, Jian |
| author_facet | Yuen, Robin Shing-Hei Tse, Timothy Tin-Long Zhu, Jian |
| contents | Current speech-based LLMs are predominantly trained on extensive ASR and TTS datasets, excelling in tasks related to these domains. However, their ability to handle direct speech-to-speech conversations remains notably constrained. These models often rely on an ASR-to-TTS chain-of-thought pipeline, converting speech into text for processing before generating audio responses, which introduces latency and loses audio features. We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions. We also release a large-scale synthetic conversational dataset to facilitate further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17353 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM Yuen, Robin Shing-Hei Tse, Timothy Tin-Long Zhu, Jian Computation and Language Current speech-based LLMs are predominantly trained on extensive ASR and TTS datasets, excelling in tasks related to these domains. However, their ability to handle direct speech-to-speech conversations remains notably constrained. These models often rely on an ASR-to-TTS chain-of-thought pipeline, converting speech into text for processing before generating audio responses, which introduces latency and loses audio features. We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions. We also release a large-scale synthetic conversational dataset to facilitate further research. |
| title | Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.17353 |