Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917575808516096 |
|---|---|
| author | Wang, Jinhan Chen, Long Khare, Aparna Raju, Anirudh Dheram, Pranav He, Di Wu, Minhua Stolcke, Andreas Ravichandran, Venkatesh |
| author_facet | Wang, Jinhan Chen, Long Khare, Aparna Raju, Anirudh Dheram, Pranav He, Di Wu, Minhua Stolcke, Andreas Ravichandran, Venkatesh |
| contents | We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM). Experiments on the Switchboard human-human conversation dataset demonstrate that our approach consistently outperforms the baseline models with single modality. We also develop a novel multi-task instruction fine-tuning strategy to further benefit from LLM-encoded knowledge for understanding the tasks and conversational contexts, leading to additional improvements. Our approach demonstrates the potential of combined LLMs and acoustic models for a more natural and conversational interaction between humans and speech-enabled AI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_14717 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion Wang, Jinhan Chen, Long Khare, Aparna Raju, Anirudh Dheram, Pranav He, Di Wu, Minhua Stolcke, Andreas Ravichandran, Venkatesh Computation and Language Artificial Intelligence Machine Learning Sound Audio and Speech Processing We propose an approach for continuous prediction of turn-taking and backchanneling locations in spoken dialogue by fusing a neural acoustic model with a large language model (LLM). Experiments on the Switchboard human-human conversation dataset demonstrate that our approach consistently outperforms the baseline models with single modality. We also develop a novel multi-task instruction fine-tuning strategy to further benefit from LLM-encoded knowledge for understanding the tasks and conversational contexts, leading to additional improvements. Our approach demonstrates the potential of combined LLMs and acoustic models for a more natural and conversational interaction between humans and speech-enabled AI agents. |
| title | Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusion |
| topic | Computation and Language Artificial Intelligence Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2401.14717 |