LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911078764511232 |
|---|---|
| author | Mo, Zhiwen Wang, Lei Wei, Jianyu Zeng, Zhichen Cao, Shijie Ma, Lingxiao Jing, Naifeng Cao, Ting Xue, Jilong Yang, Fan Yang, Mao |
| author_facet | Mo, Zhiwen Wang, Lei Wei, Jianyu Zeng, Zhichen Cao, Shijie Ma, Lingxiao Jing, Naifeng Cao, Ting Xue, Jilong Yang, Fan Yang, Mao |
| contents | Large Language Model (LLM) inference becomes resource-intensive, prompting a shift toward low-bit model weights to reduce the memory footprint and improve efficiency. Such low-bit LLMs necessitate the mixed-precision matrix multiplication (mpGEMM), an important yet underexplored operation involving the multiplication of lower-precision weights with higher-precision activations. Off-the-shelf hardware does not support this operation natively, leading to indirect, thus inefficient, dequantization-based implementations.
In this paper, we study the lookup table (LUT)-based approach for mpGEMM and find that a conventional LUT implementation fails to achieve the promised gains. To unlock the full potential of LUT-based mpGEMM, we propose LUT Tensor Core, a software-hardware co-design for low-bit LLM inference. LUT Tensor Core differentiates itself from conventional LUT designs through: 1) software-based optimizations to minimize table precompute overhead and weight reinterpretation to reduce table storage; 2) a LUT-based Tensor Core hardware design with an elongated tiling shape to maximize table reuse and a bit-serial design to support diverse precision combinations in mpGEMM; 3) a new instruction set and compilation optimizations for LUT-based mpGEMM. LUT Tensor Core significantly outperforms existing pure software LUT implementations and achieves a 1.44$\times$ improvement in compute density and energy efficiency compared to previous state-of-the-art LUT-based accelerators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_06003 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference Mo, Zhiwen Wang, Lei Wei, Jianyu Zeng, Zhichen Cao, Shijie Ma, Lingxiao Jing, Naifeng Cao, Ting Xue, Jilong Yang, Fan Yang, Mao Hardware Architecture Machine Learning C.1.0; C.3; B.2.4 Large Language Model (LLM) inference becomes resource-intensive, prompting a shift toward low-bit model weights to reduce the memory footprint and improve efficiency. Such low-bit LLMs necessitate the mixed-precision matrix multiplication (mpGEMM), an important yet underexplored operation involving the multiplication of lower-precision weights with higher-precision activations. Off-the-shelf hardware does not support this operation natively, leading to indirect, thus inefficient, dequantization-based implementations. In this paper, we study the lookup table (LUT)-based approach for mpGEMM and find that a conventional LUT implementation fails to achieve the promised gains. To unlock the full potential of LUT-based mpGEMM, we propose LUT Tensor Core, a software-hardware co-design for low-bit LLM inference. LUT Tensor Core differentiates itself from conventional LUT designs through: 1) software-based optimizations to minimize table precompute overhead and weight reinterpretation to reduce table storage; 2) a LUT-based Tensor Core hardware design with an elongated tiling shape to maximize table reuse and a bit-serial design to support diverse precision combinations in mpGEMM; 3) a new instruction set and compilation optimizations for LUT-based mpGEMM. LUT Tensor Core significantly outperforms existing pure software LUT implementations and achieves a 1.44$\times$ improvement in compute density and energy efficiency compared to previous state-of-the-art LUT-based accelerators. |
| title | LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference |
| topic | Hardware Architecture Machine Learning C.1.0; C.3; B.2.4 |
| url | https://arxiv.org/abs/2408.06003 |