CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910829625999360 |
|---|---|
| author | Chong, Zheng Dong, Xiao Li, Haoxiang Zhang, Shiyue Zhang, Wenqing Zhang, Xujie Zhao, Hanqing Jiang, Dongmei Liang, Xiaodan |
| author_facet | Chong, Zheng Dong, Xiao Li, Haoxiang Zhang, Shiyue Zhang, Wenqing Zhang, Xujie Zhao, Hanqing Jiang, Dongmei Liang, Xiaodan |
| contents | Virtual try-on methods based on diffusion models achieve realistic effects but often require additional encoding modules, a large number of training parameters, and complex preprocessing, which increases the burden on training and inference. In this work, we re-evaluate the necessity of additional modules and analyze how to improve training efficiency and reduce redundant steps in the inference process. Based on these insights, we propose CatVTON, a simple and efficient virtual try-on diffusion model that transfers in-shop or worn garments of arbitrary categories to target individuals by concatenating them along spatial dimensions as inputs of the diffusion model. The efficiency of CatVTON is reflected in three aspects: (1) Lightweight network. CatVTON consists only of a VAE and a simplified denoising UNet, removing redundant image and text encoders as well as cross-attentions, and includes just 899.06M parameters. (2) Parameter-efficient training. Through experimental analysis, we identify self-attention modules as crucial for adapting pre-trained diffusion models to the virtual try-on task, enabling high-quality results with only 49.57M training parameters. (3) Simplified inference. CatVTON eliminates unnecessary preprocessing, such as pose estimation, human parsing, and captioning, requiring only a person image and garment reference to guide the virtual try-on process, reducing over 49% memory usage compared to other diffusion-based methods. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results compared to baseline methods and demonstrates strong generalization performance in in-the-wild scenarios, despite being trained solely on public datasets with 73K samples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15886 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models Chong, Zheng Dong, Xiao Li, Haoxiang Zhang, Shiyue Zhang, Wenqing Zhang, Xujie Zhao, Hanqing Jiang, Dongmei Liang, Xiaodan Computer Vision and Pattern Recognition Artificial Intelligence 68T42 (Primary) 168T45 (Secondary) I.4.9 Virtual try-on methods based on diffusion models achieve realistic effects but often require additional encoding modules, a large number of training parameters, and complex preprocessing, which increases the burden on training and inference. In this work, we re-evaluate the necessity of additional modules and analyze how to improve training efficiency and reduce redundant steps in the inference process. Based on these insights, we propose CatVTON, a simple and efficient virtual try-on diffusion model that transfers in-shop or worn garments of arbitrary categories to target individuals by concatenating them along spatial dimensions as inputs of the diffusion model. The efficiency of CatVTON is reflected in three aspects: (1) Lightweight network. CatVTON consists only of a VAE and a simplified denoising UNet, removing redundant image and text encoders as well as cross-attentions, and includes just 899.06M parameters. (2) Parameter-efficient training. Through experimental analysis, we identify self-attention modules as crucial for adapting pre-trained diffusion models to the virtual try-on task, enabling high-quality results with only 49.57M training parameters. (3) Simplified inference. CatVTON eliminates unnecessary preprocessing, such as pose estimation, human parsing, and captioning, requiring only a person image and garment reference to guide the virtual try-on process, reducing over 49% memory usage compared to other diffusion-based methods. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results compared to baseline methods and demonstrates strong generalization performance in in-the-wild scenarios, despite being trained solely on public datasets with 73K samples. |
| title | CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence 68T42 (Primary) 168T45 (Secondary) I.4.9 |
| url | https://arxiv.org/abs/2407.15886 |