Inference Performance Optimization for Large Language Models on CPUs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917717633662976 |
|---|---|
| author | He, Pujiang Zhou, Shan Huang, Wenhuan Li, Changqing Wang, Duyi Guo, Bin Meng, Chen Gui, Sheng Yu, Weifei Xie, Yi |
| author_facet | He, Pujiang Zhou, Shan Huang, Wenhuan Li, Changqing Wang, Duyi Guo, Bin Meng, Chen Gui, Sheng Yu, Weifei Xie, Yi |
| contents | Large language models (LLMs) have shown exceptional performance and vast potential across diverse tasks. However, the deployment of LLMs with high performance in low-resource environments has garnered significant attention in the industry. When GPU hardware resources are limited, we can explore alternative options on CPUs. To mitigate the financial burden and alleviate constraints imposed by hardware resources, optimizing inference performance is necessary. In this paper, we introduce an easily deployable inference performance optimization solution aimed at accelerating LLMs on CPUs. In this solution, we implement an effective way to reduce the KV cache size while ensuring precision. We propose a distributed inference optimization approach and implement it based on oneAPI Collective Communications Library. Furthermore, we propose optimization approaches for LLMs on CPU, and conduct tailored optimizations for the most commonly used models. The code is open-sourced at https://github.com/intel/xFasterTransformer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07304 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Inference Performance Optimization for Large Language Models on CPUs He, Pujiang Zhou, Shan Huang, Wenhuan Li, Changqing Wang, Duyi Guo, Bin Meng, Chen Gui, Sheng Yu, Weifei Xie, Yi Artificial Intelligence Large language models (LLMs) have shown exceptional performance and vast potential across diverse tasks. However, the deployment of LLMs with high performance in low-resource environments has garnered significant attention in the industry. When GPU hardware resources are limited, we can explore alternative options on CPUs. To mitigate the financial burden and alleviate constraints imposed by hardware resources, optimizing inference performance is necessary. In this paper, we introduce an easily deployable inference performance optimization solution aimed at accelerating LLMs on CPUs. In this solution, we implement an effective way to reduce the KV cache size while ensuring precision. We propose a distributed inference optimization approach and implement it based on oneAPI Collective Communications Library. Furthermore, we propose optimization approaches for LLMs on CPU, and conduct tailored optimizations for the most commonly used models. The code is open-sourced at https://github.com/intel/xFasterTransformer. |
| title | Inference Performance Optimization for Large Language Models on CPUs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2407.07304 |