Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909799539539968 |
|---|---|
| author | Blandón, María Andrea Cruz Aldeneh, Zakaria Chi, Jie de Seyssel, Maureen |
| author_facet | Blandón, María Andrea Cruz Aldeneh, Zakaria Chi, Jie de Seyssel, Maureen |
| contents | Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly in monolingual settings. However, multilingual SSL models tend to underperform their monolingual counterparts on each individual language, especially in multilingual scenarios with few languages such as the bilingual setting. In this work, we investigate a novel approach to reduce this performance gap by introducing limited visual grounding into bilingual speech SSL models. Our results show that visual grounding benefits both monolingual and bilingual models, with especially pronounced gains for the latter, reducing the multilingual performance gap on zero-shot phonetic discrimination from 31.5% for audio-only models to 8.04% with grounding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17523 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models Blandón, María Andrea Cruz Aldeneh, Zakaria Chi, Jie de Seyssel, Maureen Computation and Language Audio and Speech Processing Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly in monolingual settings. However, multilingual SSL models tend to underperform their monolingual counterparts on each individual language, especially in multilingual scenarios with few languages such as the bilingual setting. In this work, we investigate a novel approach to reduce this performance gap by introducing limited visual grounding into bilingual speech SSL models. Our results show that visual grounding benefits both monolingual and bilingual models, with especially pronounced gains for the latter, reducing the multilingual performance gap on zero-shot phonetic discrimination from 31.5% for audio-only models to 8.04% with grounding. |
| title | Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2509.17523 |