Towards Multilingual Audio-Visual Question Answering
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911916096487424 |
|---|---|
| author | Phukan, Orchid Chetia Mallick, Priyabrata Behera, Swarup Ranjan Narayani, Aalekhya Satya Buduru, Arun Balaji Sharma, Rajesh |
| author_facet | Phukan, Orchid Chetia Mallick, Priyabrata Behera, Swarup Ranjan Narayani, Aalekhya Satya Buduru, Arun Balaji Sharma, Rajesh |
| contents | In this paper, we work towards extending Audio-Visual Question Answering (AVQA) to multilingual settings. Existing AVQA research has predominantly revolved around English and replicating it for addressing AVQA in other languages requires a substantial allocation of resources. As a scalable solution, we leverage machine translation and present two multilingual AVQA datasets for eight languages created from existing benchmark AVQA datasets. This prevents extra human annotation efforts of collecting questions and answers manually. To this end, we propose, MERA framework, by leveraging state-of-the-art (SOTA) video, audio, and textual foundation models for AVQA in multiple languages. We introduce a suite of models namely MERA-L, MERA-C, MERA-T with varied model architectures to benchmark the proposed datasets. We believe our work will open new research directions and act as a reference benchmark for future works in multilingual AVQA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09156 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Multilingual Audio-Visual Question Answering Phukan, Orchid Chetia Mallick, Priyabrata Behera, Swarup Ranjan Narayani, Aalekhya Satya Buduru, Arun Balaji Sharma, Rajesh Machine Learning Computer Vision and Pattern Recognition Multimedia Sound Audio and Speech Processing 68T45 In this paper, we work towards extending Audio-Visual Question Answering (AVQA) to multilingual settings. Existing AVQA research has predominantly revolved around English and replicating it for addressing AVQA in other languages requires a substantial allocation of resources. As a scalable solution, we leverage machine translation and present two multilingual AVQA datasets for eight languages created from existing benchmark AVQA datasets. This prevents extra human annotation efforts of collecting questions and answers manually. To this end, we propose, MERA framework, by leveraging state-of-the-art (SOTA) video, audio, and textual foundation models for AVQA in multiple languages. We introduce a suite of models namely MERA-L, MERA-C, MERA-T with varied model architectures to benchmark the proposed datasets. We believe our work will open new research directions and act as a reference benchmark for future works in multilingual AVQA. |
| title | Towards Multilingual Audio-Visual Question Answering |
| topic | Machine Learning Computer Vision and Pattern Recognition Multimedia Sound Audio and Speech Processing 68T45 |
| url | https://arxiv.org/abs/2406.09156 |