Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.05555 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912111382233088 |
|---|---|
| author | Bournias, Ilias Cavigelli, Lukas Zacharopoulos, Georgios |
| author_facet | Bournias, Ilias Cavigelli, Lukas Zacharopoulos, Georgios |
| contents | Large Language Model (LLM) inference on large-scale systems is expected to dominate future cloud infrastructures. Efficient LLM inference in cloud environments with numerous AI accelerators is challenging, necessitating extensive optimizations for optimal performance. Current systems batch prefill and decoding to boost throughput but encounter latency issues, while others disaggregate these phases, leading to resource underutilization. We propose AcceLLM, a novel method addressing latency and load balancing, inspired by the cache data management. It strategically utilizes redundant data to enhance inference via load balancing and optimal hardware use. Simulated evaluations on Nvidia H100 GPU and Huawei Ascend 910B2 show AcceLLM surpasses state-of-the-art systems up to 30% in latency and efficiency, handling diverse workloads effectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05555 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality Bournias, Ilias Cavigelli, Lukas Zacharopoulos, Georgios Distributed, Parallel, and Cluster Computing Large Language Model (LLM) inference on large-scale systems is expected to dominate future cloud infrastructures. Efficient LLM inference in cloud environments with numerous AI accelerators is challenging, necessitating extensive optimizations for optimal performance. Current systems batch prefill and decoding to boost throughput but encounter latency issues, while others disaggregate these phases, leading to resource underutilization. We propose AcceLLM, a novel method addressing latency and load balancing, inspired by the cache data management. It strategically utilizes redundant data to enhance inference via load balancing and optimal hardware use. Simulated evaluations on Nvidia H100 GPU and Huawei Ascend 910B2 show AcceLLM surpasses state-of-the-art systems up to 30% in latency and efficiency, handling diverse workloads effectively. |
| title | AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2411.05555 |