Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.14046 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912202401775616 |
|---|---|
| author | Palaev, Andrey Khan, Adil Kazmi, Syed M. Ahsan |
| author_facet | Palaev, Andrey Khan, Adil Kazmi, Syed M. Ahsan |
| contents | The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at https://github.com/Palandr123/DiffusionU-NetLLM |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_14046 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps Palaev, Andrey Khan, Adil Kazmi, Syed M. Ahsan Computer Vision and Pattern Recognition The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at https://github.com/Palandr123/DiffusionU-NetLLM |
| title | LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.14046 |