FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916839258324992 |
|---|---|
| author | Yuan, Tianshuo Lin, Yuxiang Wang, Jue Cheng, Zhi-Qi Wang, Xiaolong GH, Jiao Chen, Wei Peng, Xiaojiang |
| author_facet | Yuan, Tianshuo Lin, Yuxiang Wang, Jue Cheng, Zhi-Qi Wang, Xiaolong GH, Jiao Chen, Wei Peng, Xiaojiang |
| contents | Combining Vision Large Language Models (VLLMs) with diffusion models offers a powerful method for executing image editing tasks based on human language instructions. However, language instructions alone often fall short in accurately conveying user requirements, particularly when users want to add, replace elements in specific areas of an image. Luckily, masks can effectively indicate the exact locations or elements to be edited, while they require users to precisely draw the shapes at the desired locations, which is highly user-unfriendly. To address this, we propose FlexEdit, an end-to-end image editing method that leverages both free-shape masks and language instructions for Flexible Editing. Our approach employs a VLLM in comprehending the image content, mask, and user instructions. Additionally, we introduce the Mask Enhance Adapter (MEA) that fuses the embeddings of the VLLM with the image data, ensuring a seamless integration of mask information and model output embeddings. Furthermore, we construct FSMI-Edit, a benchmark specifically tailored for free-shape mask, including 8 types of free-shape mask. Extensive experiments show that our method achieves state-of-the-art (SOTA) performance in LLM-based image editing, and our simple prompting technique stands out in its effectiveness. The code and data can be found at https://github.com/A-new-b/flex_edit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12429 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editing Yuan, Tianshuo Lin, Yuxiang Wang, Jue Cheng, Zhi-Qi Wang, Xiaolong GH, Jiao Chen, Wei Peng, Xiaojiang Computer Vision and Pattern Recognition Combining Vision Large Language Models (VLLMs) with diffusion models offers a powerful method for executing image editing tasks based on human language instructions. However, language instructions alone often fall short in accurately conveying user requirements, particularly when users want to add, replace elements in specific areas of an image. Luckily, masks can effectively indicate the exact locations or elements to be edited, while they require users to precisely draw the shapes at the desired locations, which is highly user-unfriendly. To address this, we propose FlexEdit, an end-to-end image editing method that leverages both free-shape masks and language instructions for Flexible Editing. Our approach employs a VLLM in comprehending the image content, mask, and user instructions. Additionally, we introduce the Mask Enhance Adapter (MEA) that fuses the embeddings of the VLLM with the image data, ensuring a seamless integration of mask information and model output embeddings. Furthermore, we construct FSMI-Edit, a benchmark specifically tailored for free-shape mask, including 8 types of free-shape mask. Extensive experiments show that our method achieves state-of-the-art (SOTA) performance in LLM-based image editing, and our simple prompting technique stands out in its effectiveness. The code and data can be found at https://github.com/A-new-b/flex_edit. |
| title | FlexEdit: Marrying Free-Shape Masks to VLLM for Flexible Image Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.12429 |