E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916337924702208 |
|---|---|
| author | Xue, Hongfei Liang, Yuhao Mu, Bingshen Zhang, Shiliang Chen, Mengzhe Chen, Qian Xie, Lei |
| author_facet | Xue, Hongfei Liang, Yuhao Mu, Bingshen Zhang, Shiliang Chen, Mengzhe Chen, Qian Xie, Lei |
| contents | This study focuses on emotion-sensitive spoken dialogue in human-machine speech interaction. With the advancement of Large Language Models (LLMs), dialogue systems can handle multimodal data, including audio. Recent models have enhanced the understanding of complex audio signals through the integration of various audio events. However, they are unable to generate appropriate responses based on emotional speech. To address this, we introduce the Emotional chat Model (E-chat), a novel spoken dialogue system capable of comprehending and responding to emotions conveyed from speech. This model leverages an emotion embedding extracted by a speech encoder, combined with LLMs, enabling it to respond according to different emotional contexts. Additionally, we introduce the E-chat200 dataset, designed explicitly for emotion-sensitive spoken dialogue. In various evaluation metrics, E-chat consistently outperforms baseline model, demonstrating its potential in emotional comprehension and human-machine interaction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_00475 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models Xue, Hongfei Liang, Yuhao Mu, Bingshen Zhang, Shiliang Chen, Mengzhe Chen, Qian Xie, Lei Sound Audio and Speech Processing This study focuses on emotion-sensitive spoken dialogue in human-machine speech interaction. With the advancement of Large Language Models (LLMs), dialogue systems can handle multimodal data, including audio. Recent models have enhanced the understanding of complex audio signals through the integration of various audio events. However, they are unable to generate appropriate responses based on emotional speech. To address this, we introduce the Emotional chat Model (E-chat), a novel spoken dialogue system capable of comprehending and responding to emotions conveyed from speech. This model leverages an emotion embedding extracted by a speech encoder, combined with LLMs, enabling it to respond according to different emotional contexts. Additionally, we introduce the E-chat200 dataset, designed explicitly for emotion-sensitive spoken dialogue. In various evaluation metrics, E-chat consistently outperforms baseline model, demonstrating its potential in emotional comprehension and human-machine interaction. |
| title | E-chat: Emotion-sensitive Spoken Dialogue System with Large Language Models |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2401.00475 |