SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teacher
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909296901488640 |
|---|---|
| author | Dao, Trung Nguyen, Thuan Hoang Le, Thanh Vu, Duc Nguyen, Khoi Pham, Cuong Tran, Anh |
| author_facet | Dao, Trung Nguyen, Thuan Hoang Le, Thanh Vu, Duc Nguyen, Khoi Pham, Cuong Tran, Anh |
| contents | In this paper, we aim to enhance the performance of SwiftBrush, a prominent one-step text-to-image diffusion model, to be competitive with its multi-step Stable Diffusion counterpart. Initially, we explore the quality-diversity trade-off between SwiftBrush and SD Turbo: the former excels in image diversity, while the latter excels in image quality. This observation motivates our proposed modifications in the training methodology, including better weight initialization and efficient LoRA training. Moreover, our introduction of a novel clamped CLIP loss enhances image-text alignment and results in improved image quality. Remarkably, by combining the weights of models trained with efficient LoRA and full training, we achieve a new state-of-the-art one-step diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and multi-step Stable Diffusion models. The project page is available at https://swiftbrushv2.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14176 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teacher Dao, Trung Nguyen, Thuan Hoang Le, Thanh Vu, Duc Nguyen, Khoi Pham, Cuong Tran, Anh Computer Vision and Pattern Recognition Artificial Intelligence In this paper, we aim to enhance the performance of SwiftBrush, a prominent one-step text-to-image diffusion model, to be competitive with its multi-step Stable Diffusion counterpart. Initially, we explore the quality-diversity trade-off between SwiftBrush and SD Turbo: the former excels in image diversity, while the latter excels in image quality. This observation motivates our proposed modifications in the training methodology, including better weight initialization and efficient LoRA training. Moreover, our introduction of a novel clamped CLIP loss enhances image-text alignment and results in improved image quality. Remarkably, by combining the weights of models trained with efficient LoRA and full training, we achieve a new state-of-the-art one-step diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and multi-step Stable Diffusion models. The project page is available at https://swiftbrushv2.github.io. |
| title | SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teacher |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2408.14176 |