MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915326126456832 |
|---|---|
| author | Xing, Sen Zhong, Muyan Lai, Zeqiang Li, Liangchen Liu, Jiawen Wang, Yaohui Dai, Jifeng Wang, Wenhai |
| author_facet | Xing, Sen Zhong, Muyan Lai, Zeqiang Li, Liangchen Liu, Jiawen Wang, Yaohui Dai, Jifeng Wang, Wenhai |
| contents | In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages.Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01271 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost Xing, Sen Zhong, Muyan Lai, Zeqiang Li, Liangchen Liu, Jiawen Wang, Yaohui Dai, Jifeng Wang, Wenhai Computation and Language Artificial Intelligence In this work, we explore a cost-effective framework for multilingual image generation. We find that, unlike models tuned on high-quality images with multilingual annotations, leveraging text encoders pre-trained on widely available, noisy Internet image-text pairs significantly enhances data efficiency in text-to-image (T2I) generation across multiple languages.Based on this insight, we introduce MuLan, Multi-Language adapter, a lightweight language adapter with fewer than 20M parameters, trained alongside a frozen text encoder and image diffusion model. Compared to previous multilingual T2I models, this framework offers: (1) Cost efficiency. Using readily accessible English data and off-the-shelf multilingual text encoders minimizes the training cost; (2) High performance. Achieving comparable generation capabilities in over 110 languages with CLIP similarity scores nearly matching those in English (39.57 for English vs. 39.61 for other languages); and (3) Broad applicability. Seamlessly integrating with compatible community tools like LoRA, LCM, ControlNet, and IP-Adapter, expanding its potential use cases. |
| title | MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.01271 |