Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2505.21954 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909625885917184 |
|---|---|
| author | Nguyen, Le Thien Phuc Yu, Zhuoran Cao, Khoa Quang Nhat Guo, Yuwei Pham, Tu Ho Manh Nguyen, Tuan Tai Vo, Toan Ngo Duc Poon, Lucas Lee, Soochahn Lee, Yong Jae |
| author_facet | Nguyen, Le Thien Phuc Yu, Zhuoran Cao, Khoa Quang Nhat Guo, Yuwei Pham, Tu Ho Manh Nguyen, Tuan Tai Vo, Toan Ngo Duc Poon, Lucas Lee, Soochahn Lee, Yong Jae |
| contents | We present UniTalk, a novel dataset specifically designed for the task of active speaker detection, emphasizing challenging scenarios to enhance model generalization. Unlike previously established benchmarks such as AVA, which predominantly features old movies and thus exhibits significant domain gaps, UniTalk focuses explicitly on diverse and difficult real-world conditions. These include underrepresented languages, noisy backgrounds, and crowded scenes - such as multiple visible speakers speaking concurrently or in overlapping turns. It contains over 44.5 hours of video with frame-level active speaker annotations across 48,693 speaking identities, and spans a broad range of video types that reflect real-world conditions. Through rigorous evaluation, we show that state-of-the-art models, while achieving nearly perfect scores on AVA, fail to reach saturation on UniTalk, suggesting that the ASD task remains far from solved under realistic conditions. Nevertheless, models trained on UniTalk demonstrate stronger generalization to modern "in-the-wild" datasets like Talkies and ASW, as well as to AVA. UniTalk thus establishes a new benchmark for active speaker detection, providing researchers with a valuable resource for developing and evaluating versatile and resilient models.
Dataset: https://huggingface.co/datasets/plnguyen2908/UniTalk-ASD
Code: https://github.com/plnguyen2908/UniTalk-ASD-code |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21954 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniTalk: Towards Universal Active Speaker Detection in Real World Scenarios Nguyen, Le Thien Phuc Yu, Zhuoran Cao, Khoa Quang Nhat Guo, Yuwei Pham, Tu Ho Manh Nguyen, Tuan Tai Vo, Toan Ngo Duc Poon, Lucas Lee, Soochahn Lee, Yong Jae Computer Vision and Pattern Recognition Artificial Intelligence We present UniTalk, a novel dataset specifically designed for the task of active speaker detection, emphasizing challenging scenarios to enhance model generalization. Unlike previously established benchmarks such as AVA, which predominantly features old movies and thus exhibits significant domain gaps, UniTalk focuses explicitly on diverse and difficult real-world conditions. These include underrepresented languages, noisy backgrounds, and crowded scenes - such as multiple visible speakers speaking concurrently or in overlapping turns. It contains over 44.5 hours of video with frame-level active speaker annotations across 48,693 speaking identities, and spans a broad range of video types that reflect real-world conditions. Through rigorous evaluation, we show that state-of-the-art models, while achieving nearly perfect scores on AVA, fail to reach saturation on UniTalk, suggesting that the ASD task remains far from solved under realistic conditions. Nevertheless, models trained on UniTalk demonstrate stronger generalization to modern "in-the-wild" datasets like Talkies and ASW, as well as to AVA. UniTalk thus establishes a new benchmark for active speaker detection, providing researchers with a valuable resource for developing and evaluating versatile and resilient models. Dataset: https://huggingface.co/datasets/plnguyen2908/UniTalk-ASD Code: https://github.com/plnguyen2908/UniTalk-ASD-code |
| title | UniTalk: Towards Universal Active Speaker Detection in Real World Scenarios |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2505.21954 |