Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.04932 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914966573940736 |
|---|---|
| author | Li, Leheng Qiu, Weichao Yan, Xu He, Jing Zhou, Kaiqiang Cai, Yingjie Lian, Qing Liu, Bingbing Chen, Ying-Cong |
| author_facet | Li, Leheng Qiu, Weichao Yan, Xu He, Jing Zhou, Kaiqiang Cai, Yingjie Lian, Qing Liu, Bingbing Chen, Ying-Cong |
| contents | We present OmniBooth, an image generation framework that enables spatial control with instance-level multi-modal customization. For all instances, the multimodal instruction can be described through text prompts or image references. Given a set of user-defined masks and associated text or image guidance, our objective is to generate an image, where multiple objects are positioned at specified coordinates and their attributes are precisely aligned with the corresponding guidance. This approach significantly expands the scope of text-to-image generation, and elevates it to a more versatile and practical dimension in controllability. In this paper, our core contribution lies in the proposed latent control signals, a high-dimensional spatial feature that provides a unified representation to integrate the spatial, textual, and image conditions seamlessly. The text condition extends ControlNet to provide instance-level open-vocabulary generation. The image condition further enables fine-grained control with personalized identity. In practice, our method empowers users with more flexibility in controllable generation, as users can choose multi-modal conditions from text or images as needed. Furthermore, thorough experiments demonstrate our enhanced performance in image synthesis fidelity and alignment across different tasks and datasets. Project page: https://len-li.github.io/omnibooth-web/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04932 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction Li, Leheng Qiu, Weichao Yan, Xu He, Jing Zhou, Kaiqiang Cai, Yingjie Lian, Qing Liu, Bingbing Chen, Ying-Cong Computer Vision and Pattern Recognition We present OmniBooth, an image generation framework that enables spatial control with instance-level multi-modal customization. For all instances, the multimodal instruction can be described through text prompts or image references. Given a set of user-defined masks and associated text or image guidance, our objective is to generate an image, where multiple objects are positioned at specified coordinates and their attributes are precisely aligned with the corresponding guidance. This approach significantly expands the scope of text-to-image generation, and elevates it to a more versatile and practical dimension in controllability. In this paper, our core contribution lies in the proposed latent control signals, a high-dimensional spatial feature that provides a unified representation to integrate the spatial, textual, and image conditions seamlessly. The text condition extends ControlNet to provide instance-level open-vocabulary generation. The image condition further enables fine-grained control with personalized identity. In practice, our method empowers users with more flexibility in controllable generation, as users can choose multi-modal conditions from text or images as needed. Furthermore, thorough experiments demonstrate our enhanced performance in image synthesis fidelity and alignment across different tasks and datasets. Project page: https://len-li.github.io/omnibooth-web/ |
| title | OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.04932 |