Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910641282875392 |
|---|---|
| author | Nagy, Rajmund Voss, Hendric Yoon, Youngwoo Kucherenko, Taras Nikolov, Teodor Hoang-Minh, Thanh McDonnell, Rachel Kopp, Stefan Neff, Michael Henter, Gustav Eje |
| author_facet | Nagy, Rajmund Voss, Hendric Yoon, Youngwoo Kucherenko, Taras Nikolov, Teodor Hoang-Minh, Thanh McDonnell, Rachel Kopp, Stefan Neff, Michael Henter, Gustav Eje |
| contents | Current evaluation practices in speech-driven gesture generation lack standardisation and focus on aspects that are easy to measure over aspects that actually matter. This leads to a situation where it is impossible to know what is the state of the art, or to know which method works better for which purpose when comparing two publications. In this position paper, we review and give details on issues with existing gesture-generation evaluation, and present a novel proposal for remedying them. Specifically, we announce an upcoming living leaderboard to benchmark progress in conversational motion synthesis. Unlike earlier gesture-generation challenges, the leaderboard will be updated with large-scale user studies of new gesture-generation systems multiple times per year, and systems on the leaderboard can be submitted to any publication venue that their authors prefer. By evolving the leaderboard evaluation data and tasks over time, the effort can keep driving progress towards the most important end goals identified by the community. We actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated. In other words, our proposal will not only make it easier for researchers to perform good evaluations, but their collective input and contributions will also help drive the future of gesture-generation research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06327 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis Nagy, Rajmund Voss, Hendric Yoon, Youngwoo Kucherenko, Taras Nikolov, Teodor Hoang-Minh, Thanh McDonnell, Rachel Kopp, Stefan Neff, Michael Henter, Gustav Eje Human-Computer Interaction Computer Vision and Pattern Recognition Graphics Machine Learning I.3; I.2 Current evaluation practices in speech-driven gesture generation lack standardisation and focus on aspects that are easy to measure over aspects that actually matter. This leads to a situation where it is impossible to know what is the state of the art, or to know which method works better for which purpose when comparing two publications. In this position paper, we review and give details on issues with existing gesture-generation evaluation, and present a novel proposal for remedying them. Specifically, we announce an upcoming living leaderboard to benchmark progress in conversational motion synthesis. Unlike earlier gesture-generation challenges, the leaderboard will be updated with large-scale user studies of new gesture-generation systems multiple times per year, and systems on the leaderboard can be submitted to any publication venue that their authors prefer. By evolving the leaderboard evaluation data and tasks over time, the effort can keep driving progress towards the most important end goals identified by the community. We actively seek community involvement across the entire evaluation pipeline: from data and tasks for the evaluation, via tooling, to the systems evaluated. In other words, our proposal will not only make it easier for researchers to perform good evaluations, but their collective input and contributions will also help drive the future of gesture-generation research. |
| title | Towards a GENEA Leaderboard -- an Extended, Living Benchmark for Evaluating and Advancing Conversational Motion Synthesis |
| topic | Human-Computer Interaction Computer Vision and Pattern Recognition Graphics Machine Learning I.3; I.2 |
| url | https://arxiv.org/abs/2410.06327 |