T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookup
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917080207458304 |
|---|---|
| author | Wei, Jianyu Li, Qingtao Cao, Shijie Ma, Lingxiao Hao, Zixu Zhang, Yanyong Hu, Xiaoyan Cao, Ting |
| author_facet | Wei, Jianyu Li, Qingtao Cao, Shijie Ma, Lingxiao Hao, Zixu Zhang, Yanyong Hu, Xiaoyan Cao, Ting |
| contents | Large language models (LLMs) are increasingly deployed on customer devices. To support them, current devices are adopting SoCs (System on Chip) with NPUs (Neural Processing Unit) installed. Although high performance is expected, LLM inference on NPUs is slower than its CPU counterpart. The reason is that NPUs have poor performance on computations other than GEMM, like dequantization. Current works either disaggregate prefill on the NPUs and decoding on the CPUs, or put both on the NPUs but with an accuracy loss. To solve this issue, based on the insight that low-bit can enable target computation encoded within an acceptably sized table, we propose table lookup to subsume hardware operations otherwise unsupported. To realize this, we overcome the conflicting hardware behavior of prefill and decoding to design a unified table layout and tiling through (1) fused two-level table-based dequantization and (2) concurrency-hierarchy-guided tiling. Based on that, we implement the prefill phase by three-stage pipeline and map the table-lookup-based decoding to NPU's vector units. Results show 1.4x and 3.1x speedup for prefill and decoding respectively, and 84% energy savings compared to the baseline NPU methods. The code is available at https://github.com/microsoft/T-MAC/tree/main/t-man. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11248 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookup Wei, Jianyu Li, Qingtao Cao, Shijie Ma, Lingxiao Hao, Zixu Zhang, Yanyong Hu, Xiaoyan Cao, Ting Hardware Architecture Large language models (LLMs) are increasingly deployed on customer devices. To support them, current devices are adopting SoCs (System on Chip) with NPUs (Neural Processing Unit) installed. Although high performance is expected, LLM inference on NPUs is slower than its CPU counterpart. The reason is that NPUs have poor performance on computations other than GEMM, like dequantization. Current works either disaggregate prefill on the NPUs and decoding on the CPUs, or put both on the NPUs but with an accuracy loss. To solve this issue, based on the insight that low-bit can enable target computation encoded within an acceptably sized table, we propose table lookup to subsume hardware operations otherwise unsupported. To realize this, we overcome the conflicting hardware behavior of prefill and decoding to design a unified table layout and tiling through (1) fused two-level table-based dequantization and (2) concurrency-hierarchy-guided tiling. Based on that, we implement the prefill phase by three-stage pipeline and map the table-lookup-based decoding to NPU's vector units. Results show 1.4x and 3.1x speedup for prefill and decoding respectively, and 84% energy savings compared to the baseline NPU methods. The code is available at https://github.com/microsoft/T-MAC/tree/main/t-man. |
| title | T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookup |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2511.11248 |