Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908513259749376 |
|---|---|
| author | Kang, Wonjune Jia, Junteng Wu, Chunyang Zhou, Wei Lakomkin, Egor Gaur, Yashesh Sari, Leda Kim, Suyoun Li, Ke Mahadeokar, Jay Kalinli, Ozlem |
| author_facet | Kang, Wonjune Jia, Junteng Wu, Chunyang Zhou, Wei Lakomkin, Egor Gaur, Yashesh Sari, Leda Kim, Suyoun Li, Ke Mahadeokar, Jay Kalinli, Ozlem |
| contents | This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights. We utilize an end-to-end system with a speech encoder, which is trained to produce token embeddings such that the LLM's response to an expressive speech prompt is aligned with its response to a semantically matching text prompt that has also been conditioned on the user's speaking style. This framework enables the encoder to generate tokens that capture both linguistic and paralinguistic information and effectively convey them to the LLM, even when the LLM's weights remain completely frozen. To the best of our knowledge, our work is the first to explore how to induce a frozen LLM to understand more than just linguistic content from speech inputs in a general interaction setting. Experiments demonstrate that our system is able to produce higher quality and more empathetic responses to expressive speech prompts compared to several baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01162 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech Kang, Wonjune Jia, Junteng Wu, Chunyang Zhou, Wei Lakomkin, Egor Gaur, Yashesh Sari, Leda Kim, Suyoun Li, Ke Mahadeokar, Jay Kalinli, Ozlem Audio and Speech Processing Computation and Language Sound This work studies the capabilities of a large language model (LLM) to understand paralinguistic aspects of speech without fine-tuning its weights. We utilize an end-to-end system with a speech encoder, which is trained to produce token embeddings such that the LLM's response to an expressive speech prompt is aligned with its response to a semantically matching text prompt that has also been conditioned on the user's speaking style. This framework enables the encoder to generate tokens that capture both linguistic and paralinguistic information and effectively convey them to the LLM, even when the LLM's weights remain completely frozen. To the best of our knowledge, our work is the first to explore how to induce a frozen LLM to understand more than just linguistic content from speech inputs in a general interaction setting. Experiments demonstrate that our system is able to produce higher quality and more empathetic responses to expressive speech prompts compared to several baselines. |
| title | Frozen Large Language Models Can Perceive Paralinguistic Aspects of Speech |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2410.01162 |