Distributed Inference Performance Optimization for LLMs on CPUs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916307242319872 |
|---|---|
| author | He, Pujiang Zhou, Shan Li, Changqing Huang, Wenhuan Yu, Weifei Wang, Duyi Meng, Chen Gui, Sheng |
| author_facet | He, Pujiang Zhou, Shan Li, Changqing Huang, Wenhuan Yu, Weifei Wang, Duyi Meng, Chen Gui, Sheng |
| contents | Large language models (LLMs) hold tremendous potential for addressing numerous real-world challenges, yet they typically demand significant computational resources and memory. Deploying LLMs onto a resource-limited hardware device with restricted memory capacity presents considerable challenges. Distributed computing emerges as a prevalent strategy to mitigate single-node memory constraints and expedite LLM inference performance. To reduce the hardware limitation burden, we proposed an efficient distributed inference optimization solution for LLMs on CPUs. We conduct experiments with the proposed solution on 5th Gen Intel Xeon Scalable Processors, and the result shows the time per output token for the LLM with 72B parameter is 140 ms/token, much faster than the average human reading speed about 200ms per token. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_00029 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Distributed Inference Performance Optimization for LLMs on CPUs He, Pujiang Zhou, Shan Li, Changqing Huang, Wenhuan Yu, Weifei Wang, Duyi Meng, Chen Gui, Sheng Distributed, Parallel, and Cluster Computing Large language models (LLMs) hold tremendous potential for addressing numerous real-world challenges, yet they typically demand significant computational resources and memory. Deploying LLMs onto a resource-limited hardware device with restricted memory capacity presents considerable challenges. Distributed computing emerges as a prevalent strategy to mitigate single-node memory constraints and expedite LLM inference performance. To reduce the hardware limitation burden, we proposed an efficient distributed inference optimization solution for LLMs on CPUs. We conduct experiments with the proposed solution on 5th Gen Intel Xeon Scalable Processors, and the result shows the time per output token for the LLM with 72B parameter is 140 ms/token, much faster than the average human reading speed about 200ms per token. |
| title | Distributed Inference Performance Optimization for LLMs on CPUs |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2407.00029 |