AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916164978868224 |
|---|---|
| author | Tseng, Yuan Berry, Layne Chen, Yi-Ting Chiu, I-Hsiang Lin, Hsuan-Hao Liu, Max Peng, Puyuan Shih, Yi-Jen Wang, Hung-Yu Wu, Haibin Huang, Po-Yao Lai, Chun-Mao Li, Shang-Wen Harwath, David Tsao, Yu Watanabe, Shinji Mohamed, Abdelrahman Feng, Chi-Luen Lee, Hung-yi |
| author_facet | Tseng, Yuan Berry, Layne Chen, Yi-Ting Chiu, I-Hsiang Lin, Hsuan-Hao Liu, Max Peng, Puyuan Shih, Yi-Jen Wang, Hung-Yu Wu, Haibin Huang, Po-Yao Lai, Chun-Mao Li, Shang-Wen Harwath, David Tsao, Yu Watanabe, Shinji Mohamed, Abdelrahman Feng, Chi-Luen Lee, Hung-yi |
| contents | Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information. However, current models often focus on a limited set of tasks, and generalization abilities of learned representations are unclear. To this end, we propose the AV-SUPERB benchmark that enables general-purpose evaluation of unimodal audio/visual and bimodal fusion representations on 7 datasets covering 5 audio-visual tasks in speech and audio processing. We evaluate 5 recent self-supervised models and show that none of these models generalize to all tasks, emphasizing the need for future study on improving universal model performance. In addition, we show that representations may be improved with intermediate-task fine-tuning and audio event classification with AudioSet serves as a strong intermediate task. We release our benchmark with evaluation code and a model submission platform to encourage further research in audio-visual learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_10787 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models Tseng, Yuan Berry, Layne Chen, Yi-Ting Chiu, I-Hsiang Lin, Hsuan-Hao Liu, Max Peng, Puyuan Shih, Yi-Jen Wang, Hung-Yu Wu, Haibin Huang, Po-Yao Lai, Chun-Mao Li, Shang-Wen Harwath, David Tsao, Yu Watanabe, Shinji Mohamed, Abdelrahman Feng, Chi-Luen Lee, Hung-yi Audio and Speech Processing Computer Vision and Pattern Recognition Multimedia Sound Audio-visual representation learning aims to develop systems with human-like perception by utilizing correlation between auditory and visual information. However, current models often focus on a limited set of tasks, and generalization abilities of learned representations are unclear. To this end, we propose the AV-SUPERB benchmark that enables general-purpose evaluation of unimodal audio/visual and bimodal fusion representations on 7 datasets covering 5 audio-visual tasks in speech and audio processing. We evaluate 5 recent self-supervised models and show that none of these models generalize to all tasks, emphasizing the need for future study on improving universal model performance. In addition, we show that representations may be improved with intermediate-task fine-tuning and audio event classification with AudioSet serves as a strong intermediate task. We release our benchmark with evaluation code and a model submission platform to encourage further research in audio-visual learning. |
| title | AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models |
| topic | Audio and Speech Processing Computer Vision and Pattern Recognition Multimedia Sound |
| url | https://arxiv.org/abs/2309.10787 |