VGGSounder: Audio-Visual Evaluations for Foundation Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914102431973376 |
|---|---|
| author | Zverev, Daniil Wiedemer, Thaddäus Prabhu, Ameya Bethge, Matthias Brendel, Wieland Koepke, A. Sophia |
| author_facet | Zverev, Daniil Wiedemer, Thaddäus Prabhu, Ameya Bethge, Matthias Brendel, Wieland Koepke, A. Sophia |
| contents | The emergence of audio-visual foundation models underscores the importance of reliably assessing their multi-modal understanding. The VGGSound dataset is commonly used as a benchmark for evaluation audio-visual classification. However, our analysis identifies several limitations of VGGSound, including incomplete labelling, partially overlapping classes, and misaligned modalities. These lead to distorted evaluations of auditory and visual capabilities. To address these limitations, we introduce VGGSounder, a comprehensively re-annotated, multi-label test set that extends VGGSound and is specifically designed to evaluate audio-visual foundation models. VGGSounder features detailed modality annotations, enabling precise analyses of modality-specific performance. Furthermore, we reveal model limitations by analysing performance degradation when adding another input modality with our new modality confusion metric. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08237 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VGGSounder: Audio-Visual Evaluations for Foundation Models Zverev, Daniil Wiedemer, Thaddäus Prabhu, Ameya Bethge, Matthias Brendel, Wieland Koepke, A. Sophia Multimedia Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing The emergence of audio-visual foundation models underscores the importance of reliably assessing their multi-modal understanding. The VGGSound dataset is commonly used as a benchmark for evaluation audio-visual classification. However, our analysis identifies several limitations of VGGSound, including incomplete labelling, partially overlapping classes, and misaligned modalities. These lead to distorted evaluations of auditory and visual capabilities. To address these limitations, we introduce VGGSounder, a comprehensively re-annotated, multi-label test set that extends VGGSound and is specifically designed to evaluate audio-visual foundation models. VGGSounder features detailed modality annotations, enabling precise analyses of modality-specific performance. Furthermore, we reveal model limitations by analysing performance degradation when adding another input modality with our new modality confusion metric. |
| title | VGGSounder: Audio-Visual Evaluations for Foundation Models |
| topic | Multimedia Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2508.08237 |