Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916336319332352 |
|---|---|
| author | Li, Zhengang Lu, Alec Xie, Yanyue Kong, Zhenglun Sun, Mengshu Tang, Hao Xue, Zhong Jia Dong, Peiyan Ding, Caiwen Wang, Yanzhi Lin, Xue Fang, Zhenman |
| author_facet | Li, Zhengang Lu, Alec Xie, Yanyue Kong, Zhenglun Sun, Mengshu Tang, Hao Xue, Zhong Jia Dong, Peiyan Ding, Caiwen Wang, Yanzhi Lin, Xue Fang, Zhenman |
| contents | Vision transformers (ViTs) have demonstrated their superior accuracy for computer vision tasks compared to convolutional neural networks (CNNs). However, ViT models are often computation-intensive for efficient deployment on resource-limited edge devices. This work proposes Quasar-ViT, a hardware-oriented quantization-aware architecture search framework for ViTs, to design efficient ViT models for hardware implementation while preserving the accuracy. First, Quasar-ViT trains a supernet using our row-wise flexible mixed-precision quantization scheme, mixed-precision weight entanglement, and supernet layer scaling techniques. Then, it applies an efficient hardware-oriented search algorithm, integrated with hardware latency and resource modeling, to determine a series of optimal subnets from supernet under different inference latency targets. Finally, we propose a series of model-adaptive designs on the FPGA platform to support the architecture search and mitigate the gap between the theoretical computation reduction and the practical inference speedup. Our searched models achieve 101.5, 159.6, and 251.6 frames-per-second (FPS) inference speed on the AMD/Xilinx ZCU102 FPGA with 80.4%, 78.6%, and 74.9% top-1 accuracy, respectively, for the ImageNet dataset, consistently outperforming prior works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_18175 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers Li, Zhengang Lu, Alec Xie, Yanyue Kong, Zhenglun Sun, Mengshu Tang, Hao Xue, Zhong Jia Dong, Peiyan Ding, Caiwen Wang, Yanzhi Lin, Xue Fang, Zhenman Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Vision transformers (ViTs) have demonstrated their superior accuracy for computer vision tasks compared to convolutional neural networks (CNNs). However, ViT models are often computation-intensive for efficient deployment on resource-limited edge devices. This work proposes Quasar-ViT, a hardware-oriented quantization-aware architecture search framework for ViTs, to design efficient ViT models for hardware implementation while preserving the accuracy. First, Quasar-ViT trains a supernet using our row-wise flexible mixed-precision quantization scheme, mixed-precision weight entanglement, and supernet layer scaling techniques. Then, it applies an efficient hardware-oriented search algorithm, integrated with hardware latency and resource modeling, to determine a series of optimal subnets from supernet under different inference latency targets. Finally, we propose a series of model-adaptive designs on the FPGA platform to support the architecture search and mitigate the gap between the theoretical computation reduction and the practical inference speedup. Our searched models achieve 101.5, 159.6, and 251.6 frames-per-second (FPS) inference speed on the AMD/Xilinx ZCU102 FPGA with 80.4%, 78.6%, and 74.9% top-1 accuracy, respectively, for the ImageNet dataset, consistently outperforming prior works. |
| title | Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.18175 |