SparseDM: Toward Sparse Efficient Diffusion Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915248143859712 |
|---|---|
| author | Wang, Kafeng Chen, Jianfei Li, He Mi, Zhenpeng Zhu, Jun |
| author_facet | Wang, Kafeng Chen, Jianfei Li, He Mi, Zhenpeng Zhu, Jun |
| contents | Diffusion models represent a powerful family of generative models widely used for image and video generation. However, the time-consuming deployment, long inference time, and requirements on large memory hinder their applications on resource constrained devices. In this paper, we propose a method based on the improved Straight-Through Estimator to improve the deployment efficiency of diffusion models. Specifically, we add sparse masks to the Convolution and Linear layers in a pre-trained diffusion model, then transfer learn the sparse model during the fine-tuning stage and turn on the sparse masks during inference. Experimental results on a Transformer and UNet-based diffusion models demonstrate that our method reduces MACs by 50% while maintaining FID. Sparse models are accelerated by approximately 1.2x on the GPU. Under other MACs conditions, the FID is also lower than 1 compared to other methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10445 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SparseDM: Toward Sparse Efficient Diffusion Models Wang, Kafeng Chen, Jianfei Li, He Mi, Zhenpeng Zhu, Jun Machine Learning Artificial Intelligence Diffusion models represent a powerful family of generative models widely used for image and video generation. However, the time-consuming deployment, long inference time, and requirements on large memory hinder their applications on resource constrained devices. In this paper, we propose a method based on the improved Straight-Through Estimator to improve the deployment efficiency of diffusion models. Specifically, we add sparse masks to the Convolution and Linear layers in a pre-trained diffusion model, then transfer learn the sparse model during the fine-tuning stage and turn on the sparse masks during inference. Experimental results on a Transformer and UNet-based diffusion models demonstrate that our method reduces MACs by 50% while maintaining FID. Sparse models are accelerated by approximately 1.2x on the GPU. Under other MACs conditions, the FID is also lower than 1 compared to other methods. |
| title | SparseDM: Toward Sparse Efficient Diffusion Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2404.10445 |