Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909878163865600 |
|---|---|
| author | Sriratanawilai, Sukrit Thongwat, Jhayahgrit Chumpu, Romrawin Payoungkhamdee, Patomporn Nutanong, Sarana Limkonchotiwat, Peerat |
| author_facet | Sriratanawilai, Sukrit Thongwat, Jhayahgrit Chumpu, Romrawin Payoungkhamdee, Patomporn Nutanong, Sarana Limkonchotiwat, Peerat |
| contents | Vision-language models (VLMs) exhibit uneven performance across languages, a problem that is often exacerbated when the model size is reduced. While Knowledge distillation (KD) demonstrates promising results in transferring knowledge from larger to smaller VLMs, applying KD in multilingualism is an underexplored area. This paper presents a controlled empirical study of KD behavior across five distillation approaches, isolating their effects on cross-lingual representation consistency and downstream performance stability under model compression. We study five distillation formulations across CLIP and SigLIP2, and evaluate them on in-domain retrieval and out-of-domain visual QA. We find that some configurations preserve or even improve multilingual retrieval robustness despite halving model size, but others fail to maintain cross-task stability, exposing design-sensitive trade-offs that aggregate accuracy alone does not reveal. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26271 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual Sriratanawilai, Sukrit Thongwat, Jhayahgrit Chumpu, Romrawin Payoungkhamdee, Patomporn Nutanong, Sarana Limkonchotiwat, Peerat Computation and Language Vision-language models (VLMs) exhibit uneven performance across languages, a problem that is often exacerbated when the model size is reduced. While Knowledge distillation (KD) demonstrates promising results in transferring knowledge from larger to smaller VLMs, applying KD in multilingualism is an underexplored area. This paper presents a controlled empirical study of KD behavior across five distillation approaches, isolating their effects on cross-lingual representation consistency and downstream performance stability under model compression. We study five distillation formulations across CLIP and SigLIP2, and evaluate them on in-domain retrieval and out-of-domain visual QA. We find that some configurations preserve or even improve multilingual retrieval robustness despite halving model size, but others fail to maintain cross-task stability, exposing design-sensitive trade-offs that aggregate accuracy alone does not reveal. |
| title | Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.26271 |