Multi-LoRA Composition for Image Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909395273646080 |
|---|---|
| author | Zhong, Ming Shen, Yelong Wang, Shuohang Lu, Yadong Jiao, Yizhu Ouyang, Siru Yu, Donghan Han, Jiawei Chen, Weizhu |
| author_facet | Zhong, Ming Shen, Yelong Wang, Shuohang Lu, Yadong Jiao, Yizhu Ouyang, Siru Yu, Donghan Han, Jiawei Chen, Weizhu |
| contents | Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated images. Nonetheless, existing methods face challenges in effectively composing multiple LoRAs, especially as the number of LoRAs to be integrated grows, thus hindering the creation of complex imagery. In this paper, we study multi-LoRA composition through a decoding-centric perspective. We present two training-free methods: LoRA Switch, which alternates between different LoRAs at each denoising step, and LoRA Composite, which simultaneously incorporates all LoRAs to guide more cohesive image synthesis. To evaluate the proposed approaches, we establish ComposLoRA, a new comprehensive testbed as part of this research. It features a diverse range of LoRA categories with 480 composition sets. Utilizing an evaluation framework based on GPT-4V, our findings demonstrate a clear improvement in performance with our methods over the prevalent baseline, particularly evident when increasing the number of LoRAs in a composition. The code, benchmarks, LoRA weights, and all evaluation details are available on our project website: https://maszhongming.github.io/Multi-LoRA-Composition. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16843 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-LoRA Composition for Image Generation Zhong, Ming Shen, Yelong Wang, Shuohang Lu, Yadong Jiao, Yizhu Ouyang, Siru Yu, Donghan Han, Jiawei Chen, Weizhu Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Graphics Machine Learning Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models for the accurate rendition of specific elements like distinct characters or unique styles in generated images. Nonetheless, existing methods face challenges in effectively composing multiple LoRAs, especially as the number of LoRAs to be integrated grows, thus hindering the creation of complex imagery. In this paper, we study multi-LoRA composition through a decoding-centric perspective. We present two training-free methods: LoRA Switch, which alternates between different LoRAs at each denoising step, and LoRA Composite, which simultaneously incorporates all LoRAs to guide more cohesive image synthesis. To evaluate the proposed approaches, we establish ComposLoRA, a new comprehensive testbed as part of this research. It features a diverse range of LoRA categories with 480 composition sets. Utilizing an evaluation framework based on GPT-4V, our findings demonstrate a clear improvement in performance with our methods over the prevalent baseline, particularly evident when increasing the number of LoRAs in a composition. The code, benchmarks, LoRA weights, and all evaluation details are available on our project website: https://maszhongming.github.io/Multi-LoRA-Composition. |
| title | Multi-LoRA Composition for Image Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Graphics Machine Learning |
| url | https://arxiv.org/abs/2402.16843 |