PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915174525435904 |
|---|---|
| author | Lin, Weifeng Wei, Xinyu Zhang, Renrui Zhuo, Le Zhao, Shitian Huang, Siyuan Teng, Huan Xie, Junlin Qiao, Yu Gao, Peng Li, Hongsheng |
| author_facet | Lin, Weifeng Wei, Xinyu Zhang, Renrui Zhuo, Le Zhao, Shitian Huang, Siyuan Teng, Huan Xie, Junlin Qiao, Yu Gao, Peng Li, Hongsheng |
| contents | This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions. The code and related resources are available at https://github.com/AFeng-x/PixWizard |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15278 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions Lin, Weifeng Wei, Xinyu Zhang, Renrui Zhuo, Le Zhao, Shitian Huang, Siyuan Teng, Huan Xie, Junlin Qiao, Yu Gao, Peng Li, Hongsheng Computer Vision and Pattern Recognition This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions. The code and related resources are available at https://github.com/AFeng-x/PixWizard |
| title | PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.15278 |