Aya Vision: Advancing the Frontier of Multilingual Multimodality
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912373939372032 |
|---|---|
| author | Dash, Saurabh Nan, Yiyang Dang, John Ahmadian, Arash Singh, Shivalika Smith, Madeline Venkitesh, Bharat Shmyhlo, Vlad Aryabumi, Viraat Beller-Morales, Walter Pekmez, Jeremy Ozuzu, Jason Richemond, Pierre Locatelli, Acyr Frosst, Nick Blunsom, Phil Gomez, Aidan Zhang, Ivan Fadaee, Marzieh Govindassamy, Manoj Roy, Sudip Gallé, Matthias Ermis, Beyza Üstün, Ahmet Hooker, Sara |
| author_facet | Dash, Saurabh Nan, Yiyang Dang, John Ahmadian, Arash Singh, Shivalika Smith, Madeline Venkitesh, Bharat Shmyhlo, Vlad Aryabumi, Viraat Beller-Morales, Walter Pekmez, Jeremy Ozuzu, Jason Richemond, Pierre Locatelli, Acyr Frosst, Nick Blunsom, Phil Gomez, Aidan Zhang, Ivan Fadaee, Marzieh Govindassamy, Manoj Roy, Sudip Gallé, Matthias Ermis, Beyza Üstün, Ahmet Hooker, Sara |
| contents | Building multimodal language models is fundamentally challenging: it requires aligning vision and language modalities, curating high-quality instruction data, and avoiding the degradation of existing text-only capabilities once vision is introduced. These difficulties are further magnified in the multilingual setting, where the need for multimodal data in different languages exacerbates existing data scarcity, machine translation often distorts meaning, and catastrophic forgetting is more pronounced. To address the aforementioned challenges, we introduce novel techniques spanning both data and modeling. First, we develop a synthetic annotation framework that curates high-quality, diverse multilingual multimodal instruction data, enabling Aya Vision models to produce natural, human-preferred responses to multimodal inputs across many languages. Complementing this, we propose a cross-modal model merging technique that mitigates catastrophic forgetting, effectively preserving text-only capabilities while simultaneously enhancing multimodal generative performance. Aya-Vision-8B achieves best-in-class performance compared to strong multimodal models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which outperforms models more than twice its size, such as Molmo-72B and LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the multi-modal frontier, and provides insights into techniques that effectively bend the need for compute while delivering extremely high performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_08751 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Aya Vision: Advancing the Frontier of Multilingual Multimodality Dash, Saurabh Nan, Yiyang Dang, John Ahmadian, Arash Singh, Shivalika Smith, Madeline Venkitesh, Bharat Shmyhlo, Vlad Aryabumi, Viraat Beller-Morales, Walter Pekmez, Jeremy Ozuzu, Jason Richemond, Pierre Locatelli, Acyr Frosst, Nick Blunsom, Phil Gomez, Aidan Zhang, Ivan Fadaee, Marzieh Govindassamy, Manoj Roy, Sudip Gallé, Matthias Ermis, Beyza Üstün, Ahmet Hooker, Sara Computation and Language Computer Vision and Pattern Recognition Machine Learning Building multimodal language models is fundamentally challenging: it requires aligning vision and language modalities, curating high-quality instruction data, and avoiding the degradation of existing text-only capabilities once vision is introduced. These difficulties are further magnified in the multilingual setting, where the need for multimodal data in different languages exacerbates existing data scarcity, machine translation often distorts meaning, and catastrophic forgetting is more pronounced. To address the aforementioned challenges, we introduce novel techniques spanning both data and modeling. First, we develop a synthetic annotation framework that curates high-quality, diverse multilingual multimodal instruction data, enabling Aya Vision models to produce natural, human-preferred responses to multimodal inputs across many languages. Complementing this, we propose a cross-modal model merging technique that mitigates catastrophic forgetting, effectively preserving text-only capabilities while simultaneously enhancing multimodal generative performance. Aya-Vision-8B achieves best-in-class performance compared to strong multimodal models such as Qwen-2.5-VL-7B, Pixtral-12B, and even much larger Llama-3.2-90B-Vision. We further scale this approach with Aya-Vision-32B, which outperforms models more than twice its size, such as Molmo-72B and LLaMA-3.2-90B-Vision. Our work advances multilingual progress on the multi-modal frontier, and provides insights into techniques that effectively bend the need for compute while delivering extremely high performance. |
| title | Aya Vision: Advancing the Frontier of Multilingual Multimodality |
| topic | Computation and Language Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2505.08751 |