LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915035479015424 |
|---|---|
| author | Xie, Rui Zhao, Tianchen Yuan, Zhihang Wan, Rui Gao, Wenxi Zhu, Zhenhua Ning, Xuefei Wang, Yu |
| author_facet | Xie, Rui Zhao, Tianchen Yuan, Zhihang Wan, Rui Gao, Wenxi Zhu, Zhenhua Ning, Xuefei Wang, Yu |
| contents | Visual Autoregressive (VAR) has emerged as a promising approach in image generation, offering competitive potential and performance comparable to diffusion-based models. However, current AR-based visual generation models require substantial computational resources, limiting their applicability on resource-constrained devices. To address this issue, we conducted analysis and identified significant redundancy in three dimensions of the VAR model: (1) the attention map, (2) the attention outputs when using classifier free guidance, and (3) the data precision. Correspondingly, we proposed efficient attention mechanism and low-bit quantization method to enhance the efficiency of VAR models while maintaining performance. With negligible performance lost (less than 0.056 FID increase), we could achieve 85.2% reduction in attention computation, 50% reduction in overall memory and 1.5x latency reduction. To ensure deployment feasibility, we developed efficient training-free compression techniques and analyze the deployment feasibility and efficiency gain of each technique. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17178 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization Xie, Rui Zhao, Tianchen Yuan, Zhihang Wan, Rui Gao, Wenxi Zhu, Zhenhua Ning, Xuefei Wang, Yu Computer Vision and Pattern Recognition Visual Autoregressive (VAR) has emerged as a promising approach in image generation, offering competitive potential and performance comparable to diffusion-based models. However, current AR-based visual generation models require substantial computational resources, limiting their applicability on resource-constrained devices. To address this issue, we conducted analysis and identified significant redundancy in three dimensions of the VAR model: (1) the attention map, (2) the attention outputs when using classifier free guidance, and (3) the data precision. Correspondingly, we proposed efficient attention mechanism and low-bit quantization method to enhance the efficiency of VAR models while maintaining performance. With negligible performance lost (less than 0.056 FID increase), we could achieve 85.2% reduction in attention computation, 50% reduction in overall memory and 1.5x latency reduction. To ensure deployment feasibility, we developed efficient training-free compression techniques and analyze the deployment feasibility and efficiency gain of each technique. |
| title | LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.17178 |