FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913358238711808 |
|---|---|
| author | He, Xuehai Zheng, Jian Fang, Jacob Zhiyuan Piramuthu, Robinson Bansal, Mohit Ordonez, Vicente Sigurdsson, Gunnar A Peng, Nanyun Wang, Xin Eric |
| author_facet | He, Xuehai Zheng, Jian Fang, Jacob Zhiyuan Piramuthu, Robinson Bansal, Mohit Ordonez, Vicente Sigurdsson, Gunnar A Peng, Nanyun Wang, Xin Eric |
| contents | Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps. Nevertheless, current controllable T2I methods commonly face challenges related to efficiency and faithfulness, especially when conditioning on multiple inputs from either the same or diverse modalities. In this paper, we propose a novel Flexible and Efficient method, FlexEControl, for controllable T2I generation. At the core of FlexEControl is a unique weight decomposition strategy, which allows for streamlined integration of various input types. This approach not only enhances the faithfulness of the generated image to the control, but also significantly reduces the computational overhead typically associated with multimodal conditioning. Our approach achieves a reduction of 41% in trainable parameters and 30% in memory usage compared with Uni-ControlNet. Moreover, it doubles data efficiency and can flexibly generate images under the guidance of multiple input conditions of various modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04834 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation He, Xuehai Zheng, Jian Fang, Jacob Zhiyuan Piramuthu, Robinson Bansal, Mohit Ordonez, Vicente Sigurdsson, Gunnar A Peng, Nanyun Wang, Xin Eric Computer Vision and Pattern Recognition Controllable text-to-image (T2I) diffusion models generate images conditioned on both text prompts and semantic inputs of other modalities like edge maps. Nevertheless, current controllable T2I methods commonly face challenges related to efficiency and faithfulness, especially when conditioning on multiple inputs from either the same or diverse modalities. In this paper, we propose a novel Flexible and Efficient method, FlexEControl, for controllable T2I generation. At the core of FlexEControl is a unique weight decomposition strategy, which allows for streamlined integration of various input types. This approach not only enhances the faithfulness of the generated image to the control, but also significantly reduces the computational overhead typically associated with multimodal conditioning. Our approach achieves a reduction of 41% in trainable parameters and 30% in memory usage compared with Uni-ControlNet. Moreover, it doubles data efficiency and can flexibly generate images under the guidance of multiple input conditions of various modalities. |
| title | FlexEControl: Flexible and Efficient Multimodal Control for Text-to-Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.04834 |