ReactMotion: Generating Reactive Listener Motions from Speaker Utterance
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917346810003456 |
|---|---|
| author | Luo, Cheng Wu, Bizhu Li, Bing Ren, Jianfeng Bai, Ruibin Qu, Rong Shen, Linlin Ghanem, Bernard |
| author_facet | Luo, Cheng Wu, Bizhu Li, Bing Ren, Jianfeng Bai, Ruibin Qu, Rong Shen, Linlin Ghanem, Bernard |
| contents | In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15083 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ReactMotion: Generating Reactive Listener Motions from Speaker Utterance Luo, Cheng Wu, Bizhu Li, Bing Ren, Jianfeng Bai, Ruibin Qu, Rong Shen, Linlin Ghanem, Bernard Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction Multimedia Sound In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions. |
| title | ReactMotion: Generating Reactive Listener Motions from Speaker Utterance |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction Multimedia Sound |
| url | https://arxiv.org/abs/2603.15083 |