Modeling Turn-Taking with Semantically Informed Gestures
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917353837559808 |
|---|---|
| author | Suresh, Varsha Mughal, M. Hamza Theobalt, Christian Demberg, Vera |
| author_facet | Suresh, Varsha Mughal, M. Hamza Theobalt, Christian Demberg, Vera |
| contents | In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19350 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Modeling Turn-Taking with Semantically Informed Gestures Suresh, Varsha Mughal, M. Hamza Theobalt, Christian Demberg, Vera Computation and Language In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking. |
| title | Modeling Turn-Taking with Semantically Informed Gestures |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.19350 |