Modeling Turn-Taking with Semantically Informed Gestures
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917353837559808 |
|---|---|
| author | Suresh, Varsha Mughal, M. Hamza Theobalt, Christian Demberg, Vera |
| author_facet | Suresh, Varsha Mughal, M. Hamza Theobalt, Christian Demberg, Vera |
| contents | In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19350 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Modeling Turn-Taking with Semantically Informed Gestures Suresh, Varsha Mughal, M. Hamza Theobalt, Christian Demberg, Vera Computation and Language In conversation, humans use multimodal cues, such as speech, gestures, and gaze, to manage turn-taking. While linguistic and acoustic features are informative, gestures provide complementary cues for modeling these transitions. To study this, we introduce DnD Gesture++, an extension of the multi-party DnD Gesture corpus enriched with 2,663 semantic gesture annotations spanning iconic, metaphoric, deictic, and discourse types. Using this dataset, we model turn-taking prediction through a Mixture-of-Experts framework integrating text, audio, and gestures. Experiments show that incorporating semantically guided gestures yields consistent performance gains over baselines, demonstrating their complementary role in multimodal turn-taking. |
| title | Modeling Turn-Taking with Semantically Informed Gestures |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.19350 |