SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909466545356800 |
|---|---|
| author | Fan, Zichen Dai, Steve Venkatesan, Rangharajan Sylvester, Dennis Khailany, Brucek |
| author_facet | Fan, Zichen Dai, Steve Venkatesan, Rangharajan Sylvester, Dennis Khailany, Brucek |
| contents | Diffusion models have gained significant popularity in image generation tasks. However, generating high-quality content remains notably slow because it requires running model inference over many time steps. To accelerate these models, we propose to aggressively quantize both weights and activations, while simultaneously promoting significant activation sparsity. We further observe that the stated sparsity pattern varies among different channels and evolves across time steps. To support this quantization and sparsity scheme, we present a novel diffusion model accelerator featuring a heterogeneous mixed-precision dense-sparse architecture, channel-last address mapping, and a time-step-aware sparsity detector for efficient handling of the sparsity pattern. Our 4-bit quantization technique demonstrates superior generation quality compared to existing 4-bit methods. Our custom accelerator achieves 6.91x speed-up and 51.5% energy reduction compared to traditional dense accelerators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15448 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity Fan, Zichen Dai, Steve Venkatesan, Rangharajan Sylvester, Dennis Khailany, Brucek Computer Vision and Pattern Recognition Artificial Intelligence Hardware Architecture Machine Learning Diffusion models have gained significant popularity in image generation tasks. However, generating high-quality content remains notably slow because it requires running model inference over many time steps. To accelerate these models, we propose to aggressively quantize both weights and activations, while simultaneously promoting significant activation sparsity. We further observe that the stated sparsity pattern varies among different channels and evolves across time steps. To support this quantization and sparsity scheme, we present a novel diffusion model accelerator featuring a heterogeneous mixed-precision dense-sparse architecture, channel-last address mapping, and a time-step-aware sparsity detector for efficient handling of the sparsity pattern. Our 4-bit quantization technique demonstrates superior generation quality compared to existing 4-bit methods. Our custom accelerator achieves 6.91x speed-up and 51.5% energy reduction compared to traditional dense accelerators. |
| title | SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Hardware Architecture Machine Learning |
| url | https://arxiv.org/abs/2501.15448 |