Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909617366237184 |
|---|---|
| author | Pikabea, Iñigo Lacunza, Iñaki Pareras, Oriol Escolano, Carlos Gonzalez-Agirre, Aitor Hernando, Javier Villegas, Marta |
| author_facet | Pikabea, Iñigo Lacunza, Iñaki Pareras, Oriol Escolano, Carlos Gonzalez-Agirre, Aitor Hernando, Javier Villegas, Marta |
| contents | Rapid advancements in Visual Language Models (VLMs) have transformed multimodal understanding but are often constrained by generating English responses regardless of the input language. This phenomenon has been termed as Image-induced Fidelity Loss (IFL) and stems from limited multimodal multilingual training data. To address this, we propose a continuous multilingual integration strategy that injects text-only multilingual data during visual instruction tuning, preserving the language model's original multilingual capabilities. Extensive evaluations demonstrate that our approach significantly improves linguistic fidelity across languages without degradation in visual performance. We also explore model merging, which improves language fidelity but comes at the cost of visual performance. In contrast, our core method achieves robust multilingual alignment without trade-offs, offering a scalable and effective path to mitigating IFL for global VLM adoption. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_22577 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization Pikabea, Iñigo Lacunza, Iñaki Pareras, Oriol Escolano, Carlos Gonzalez-Agirre, Aitor Hernando, Javier Villegas, Marta Computer Vision and Pattern Recognition Artificial Intelligence Rapid advancements in Visual Language Models (VLMs) have transformed multimodal understanding but are often constrained by generating English responses regardless of the input language. This phenomenon has been termed as Image-induced Fidelity Loss (IFL) and stems from limited multimodal multilingual training data. To address this, we propose a continuous multilingual integration strategy that injects text-only multilingual data during visual instruction tuning, preserving the language model's original multilingual capabilities. Extensive evaluations demonstrate that our approach significantly improves linguistic fidelity across languages without degradation in visual performance. We also explore model merging, which improves language fidelity but comes at the cost of visual performance. In contrast, our core method achieves robust multilingual alignment without trade-offs, offering a scalable and effective path to mitigating IFL for global VLM adoption. |
| title | Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2503.22577 |