Image Inpainting Models are Effective Tools for Instruction-guided Image Editing
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913435083603968 |
|---|---|
| author | Ju, Xuan Zhuang, Junhao Zhang, Zhaoyang Bian, Yuxuan Xu, Qiang Shan, Ying |
| author_facet | Ju, Xuan Zhuang, Junhao Zhang, Zhaoyang Bian, Yuxuan Xu, Qiang Shan, Ying |
| contents | This is the technique report for the winning solution of the CVPR2024 GenAI Media Generation Challenge Workshop's Instruction-guided Image Editing track. Instruction-guided image editing has been largely studied in recent years. The most advanced methods, such as SmartEdit and MGIE, usually combine large language models with diffusion models through joint training, where the former provides text understanding ability, and the latter provides image generation ability. However, in our experiments, we find that simply connecting large language models and image generation models through intermediary guidance such as masks instead of joint fine-tuning leads to a better editing performance and success rate. We use a 4-step process IIIE (Inpainting-based Instruction-guided Image Editing): editing category classification, main editing object identification, editing mask acquisition, and image inpainting. Results show that through proper combinations of language models and image inpainting models, our pipeline can reach a high success rate with satisfying visual quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_13139 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Image Inpainting Models are Effective Tools for Instruction-guided Image Editing Ju, Xuan Zhuang, Junhao Zhang, Zhaoyang Bian, Yuxuan Xu, Qiang Shan, Ying Computer Vision and Pattern Recognition This is the technique report for the winning solution of the CVPR2024 GenAI Media Generation Challenge Workshop's Instruction-guided Image Editing track. Instruction-guided image editing has been largely studied in recent years. The most advanced methods, such as SmartEdit and MGIE, usually combine large language models with diffusion models through joint training, where the former provides text understanding ability, and the latter provides image generation ability. However, in our experiments, we find that simply connecting large language models and image generation models through intermediary guidance such as masks instead of joint fine-tuning leads to a better editing performance and success rate. We use a 4-step process IIIE (Inpainting-based Instruction-guided Image Editing): editing category classification, main editing object identification, editing mask acquisition, and image inpainting. Results show that through proper combinations of language models and image inpainting models, our pipeline can reach a high success rate with satisfying visual quality. |
| title | Image Inpainting Models are Effective Tools for Instruction-guided Image Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.13139 |