BeLLMan: Controlling LLM Congestion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918162451136512 |
|---|---|
| author | Reddy, Tella Rajashekhar Deshmukh, Atharva Tandon, Karan Gandhi, Rohan Parayil, Anjaly Bhattacherjee, Debopam |
| author_facet | Reddy, Tella Rajashekhar Deshmukh, Atharva Tandon, Karan Gandhi, Rohan Parayil, Anjaly Bhattacherjee, Debopam |
| contents | Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferencing latency inflation and poor user experience. Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load. On a real testbed with H100 GPUs, beLLMan helps keep inferencing latency under control (upto 8X lower end-to-end latency) and reduces energy consumption by 25% (while serving 19% more requests) during periods of congestion for a summarization workload. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15330 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BeLLMan: Controlling LLM Congestion Reddy, Tella Rajashekhar Deshmukh, Atharva Tandon, Karan Gandhi, Rohan Parayil, Anjaly Bhattacherjee, Debopam Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Networking and Internet Architecture Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferencing latency inflation and poor user experience. Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load. On a real testbed with H100 GPUs, beLLMan helps keep inferencing latency under control (upto 8X lower end-to-end latency) and reduces energy consumption by 25% (while serving 19% more requests) during periods of congestion for a summarization workload. |
| title | BeLLMan: Controlling LLM Congestion |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Networking and Internet Architecture |
| url | https://arxiv.org/abs/2510.15330 |