Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918460406104064 |
|---|---|
| author | Nagy, Rajmund Voss, Hendric Hoang-Minh, Thanh Tsakov, Mihail Nikolov, Teodor Zhang, Zeyi Ao, Tenglong Yang, Sicheng Huang, Shaoli Cheng, Yongkang Mughal, M. Hamza Dabral, Rishabh Chhatre, Kiran Theobalt, Christian Liu, Libin Kopp, Stefan McDonnell, Rachel Neff, Michael Kucherenko, Taras Yoon, Youngwoo Henter, Gustav Eje |
| author_facet | Nagy, Rajmund Voss, Hendric Hoang-Minh, Thanh Tsakov, Mihail Nikolov, Teodor Zhang, Zeyi Ao, Tenglong Yang, Sicheng Huang, Shaoli Cheng, Yongkang Mughal, M. Hamza Dabral, Rishabh Chhatre, Kiran Theobalt, Christian Liu, Libin Kopp, Stefan McDonnell, Rachel Neff, Michael Kucherenko, Taras Yoon, Youngwoo Henter, Gustav Eje |
| contents | We review human evaluation practices in automatic, speech-driven 3D gesture generation and find a lack of standardisation and frequent use of flawed experimental setups. This leads to a situation where it is impossible to know how different methods compare, or what the state of the art is. In order to address common shortcomings of evaluation design, and to standardise future user studies in gesture-generation works, we introduce a detailed human evaluation protocol for the widely-used BEAT2 motion-capture dataset. Using this protocol, we conduct large-scale crowdsourced evaluation to rank six recent gesture-generation models -- each trained by its original authors -- across two key evaluation dimensions: motion realism and speech-gesture alignment. Our results show that 1) motion realism has become a saturated evaluation measure on the BEAT2 dataset, with older models performing on par with more recent approaches; 2) previous findings of high speech-gesture alignment do not hold up under rigorous evaluation, even for specialised models; and 3) the field must adopt disentangled assessments of motion quality and multimodal alignment for accurate benchmarking in order to make progress. To drive standardisation and enable new evaluation research, we release five hours of synthetic motion from the benchmarked models; over 750 rendered video stimuli from the user studies -- enabling new evaluations without requiring model reimplementation -- alongside our open-source rendering script, and 16,000 pairwise human preference votes collected for our benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_01233 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark Nagy, Rajmund Voss, Hendric Hoang-Minh, Thanh Tsakov, Mihail Nikolov, Teodor Zhang, Zeyi Ao, Tenglong Yang, Sicheng Huang, Shaoli Cheng, Yongkang Mughal, M. Hamza Dabral, Rishabh Chhatre, Kiran Theobalt, Christian Liu, Libin Kopp, Stefan McDonnell, Rachel Neff, Michael Kucherenko, Taras Yoon, Youngwoo Henter, Gustav Eje Computer Vision and Pattern Recognition Graphics Human-Computer Interaction I.3; I.2 We review human evaluation practices in automatic, speech-driven 3D gesture generation and find a lack of standardisation and frequent use of flawed experimental setups. This leads to a situation where it is impossible to know how different methods compare, or what the state of the art is. In order to address common shortcomings of evaluation design, and to standardise future user studies in gesture-generation works, we introduce a detailed human evaluation protocol for the widely-used BEAT2 motion-capture dataset. Using this protocol, we conduct large-scale crowdsourced evaluation to rank six recent gesture-generation models -- each trained by its original authors -- across two key evaluation dimensions: motion realism and speech-gesture alignment. Our results show that 1) motion realism has become a saturated evaluation measure on the BEAT2 dataset, with older models performing on par with more recent approaches; 2) previous findings of high speech-gesture alignment do not hold up under rigorous evaluation, even for specialised models; and 3) the field must adopt disentangled assessments of motion quality and multimodal alignment for accurate benchmarking in order to make progress. To drive standardisation and enable new evaluation research, we release five hours of synthetic motion from the benchmarked models; over 750 rendered video stimuli from the user studies -- enabling new evaluations without requiring model reimplementation -- alongside our open-source rendering script, and 16,000 pairwise human preference votes collected for our benchmark. |
| title | Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark |
| topic | Computer Vision and Pattern Recognition Graphics Human-Computer Interaction I.3; I.2 |
| url | https://arxiv.org/abs/2511.01233 |