Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Services
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914496692355072 |
|---|---|
| author | Chen, Haoyu Li, Xue Qian, Kun Guan, Yu Zhao, Jin Wang, Xin |
| author_facet | Chen, Haoyu Li, Xue Qian, Kun Guan, Yu Zhao, Jin Wang, Xin |
| contents | In Large Language Model (LLM) inference services, it is challenging to make a parallelism strategy configuration, to efficiently process the requests of variance context lengths. Requests of long context require high degree of parallelism to provide more memory for Key-Value (KV) Cache, while requests of short context prefer low degree of parallelism to increase concurrency, thus improving throughput. To maintain high throughput while supporting large context lengths on demand, we propose Amoeba, a runtime Tensor Parallel (TP) transformation for online LLM inference services, which adaptively adjusts the TP of running instances to align with the dynamics of incoming requests. Evaluations using real-world traces show that Amoeba improves throughput by 1.75x-6.57x compared to state-of-the-art solutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19729 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Services Chen, Haoyu Li, Xue Qian, Kun Guan, Yu Zhao, Jin Wang, Xin Distributed, Parallel, and Cluster Computing In Large Language Model (LLM) inference services, it is challenging to make a parallelism strategy configuration, to efficiently process the requests of variance context lengths. Requests of long context require high degree of parallelism to provide more memory for Key-Value (KV) Cache, while requests of short context prefer low degree of parallelism to increase concurrency, thus improving throughput. To maintain high throughput while supporting large context lengths on demand, we propose Amoeba, a runtime Tensor Parallel (TP) transformation for online LLM inference services, which adaptively adjusts the TP of running instances to align with the dynamics of incoming requests. Evaluations using real-world traces show that Amoeba improves throughput by 1.75x-6.57x compared to state-of-the-art solutions. |
| title | Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Services |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2509.19729 |