HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929262698692608 |
|---|---|
| author | Zhao, Xuanlei Jia, Bin Zhou, Haotian Liu, Ziming Cheng, Shenggan You, Yang |
| author_facet | Zhao, Xuanlei Jia, Bin Zhou, Haotian Liu, Ziming Cheng, Shenggan You, Yang |
| contents | In recent times, the emergence of Large Language Models (LLMs) has resulted in increasingly larger model size, posing challenges for inference on low-resource devices. Prior approaches have explored offloading to facilitate low-memory inference but often suffer from efficiency due to I/O bottlenecks. To achieve low-latency LLMs inference on resource-constrained devices, we introduce HeteGen, a novel approach that presents a principled framework for heterogeneous parallel computing using CPUs and GPUs. Based on this framework, HeteGen further employs heterogeneous parallel computing and asynchronous overlap for LLMs to mitigate I/O bottlenecks. Our experiments demonstrate a substantial improvement in inference speed, surpassing state-of-the-art methods by over 317% at most. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01164 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices Zhao, Xuanlei Jia, Bin Zhou, Haotian Liu, Ziming Cheng, Shenggan You, Yang Performance Distributed, Parallel, and Cluster Computing In recent times, the emergence of Large Language Models (LLMs) has resulted in increasingly larger model size, posing challenges for inference on low-resource devices. Prior approaches have explored offloading to facilitate low-memory inference but often suffer from efficiency due to I/O bottlenecks. To achieve low-latency LLMs inference on resource-constrained devices, we introduce HeteGen, a novel approach that presents a principled framework for heterogeneous parallel computing using CPUs and GPUs. Based on this framework, HeteGen further employs heterogeneous parallel computing and asynchronous overlap for LLMs to mitigate I/O bottlenecks. Our experiments demonstrate a substantial improvement in inference speed, surpassing state-of-the-art methods by over 317% at most. |
| title | HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices |
| topic | Performance Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2403.01164 |