Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.05674 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917633817837568 |
|---|---|
| author | Song, Kunpeng Zhu, Yizhe Liu, Bingchen Yan, Qing Elgammal, Ahmed Yang, Xiao |
| author_facet | Song, Kunpeng Zhu, Yizhe Liu, Bingchen Yan, Qing Elgammal, Ahmed Yang, Xiao |
| contents | In this paper, we present MoMA: an open-vocabulary, training-free personalized image model that boasts flexible zero-shot capabilities. As foundational text-to-image models rapidly evolve, the demand for robust image-to-image translation grows. Addressing this need, MoMA specializes in subject-driven personalized image generation. Utilizing an open-source, Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as both a feature extractor and a generator. This approach effectively synergizes reference image and text prompt information to produce valuable image features, facilitating an image diffusion model. To better leverage the generated features, we further introduce a novel self-attention shortcut method that efficiently transfers image features to an image diffusion model, improving the resemblance of the target object in generated images. Remarkably, as a tuning-free plug-and-play module, our model requires only a single reference image and outperforms existing methods in generating images with high detail fidelity, enhanced identity-preservation and prompt faithfulness. Our work is open-source, thereby providing universal access to these advancements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_05674 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation Song, Kunpeng Zhu, Yizhe Liu, Bingchen Yan, Qing Elgammal, Ahmed Yang, Xiao Computer Vision and Pattern Recognition In this paper, we present MoMA: an open-vocabulary, training-free personalized image model that boasts flexible zero-shot capabilities. As foundational text-to-image models rapidly evolve, the demand for robust image-to-image translation grows. Addressing this need, MoMA specializes in subject-driven personalized image generation. Utilizing an open-source, Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as both a feature extractor and a generator. This approach effectively synergizes reference image and text prompt information to produce valuable image features, facilitating an image diffusion model. To better leverage the generated features, we further introduce a novel self-attention shortcut method that efficiently transfers image features to an image diffusion model, improving the resemblance of the target object in generated images. Remarkably, as a tuning-free plug-and-play module, our model requires only a single reference image and outperforms existing methods in generating images with high detail fidelity, enhanced identity-preservation and prompt faithfulness. Our work is open-source, thereby providing universal access to these advancements. |
| title | MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.05674 |