Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909154971484160 |
|---|---|
| author | Stojkovic, Jovan Choukse, Esha Zhang, Chaojie Goiri, Inigo Torrellas, Josep |
| author_facet | Stojkovic, Jovan Choukse, Esha Zhang, Chaojie Goiri, Inigo Torrellas, Josep |
| contents | With the ubiquitous use of modern large language models (LLMs) across industries, the inference serving for these models is ever expanding. Given the high compute and memory requirements of modern LLMs, more and more top-of-the-line GPUs are being deployed to serve these models. Energy availability has come to the forefront as the biggest challenge for data center expansion to serve these models. In this paper, we present the trade-offs brought up by making energy efficiency the primary goal of LLM serving under performance SLOs. We show that depending on the inputs, the model, and the service-level agreements, there are several knobs available to the LLM inference provider to use for being energy efficient. We characterize the impact of these knobs on the latency, throughput, as well as the energy. By exploring these trade-offs, we offer valuable insights into optimizing energy usage without compromising on performance, thereby paving the way for sustainable and cost-effective LLM deployment in data center environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_20306 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference Stojkovic, Jovan Choukse, Esha Zhang, Chaojie Goiri, Inigo Torrellas, Josep Artificial Intelligence Hardware Architecture Distributed, Parallel, and Cluster Computing C.0; I.2 With the ubiquitous use of modern large language models (LLMs) across industries, the inference serving for these models is ever expanding. Given the high compute and memory requirements of modern LLMs, more and more top-of-the-line GPUs are being deployed to serve these models. Energy availability has come to the forefront as the biggest challenge for data center expansion to serve these models. In this paper, we present the trade-offs brought up by making energy efficiency the primary goal of LLM serving under performance SLOs. We show that depending on the inputs, the model, and the service-level agreements, there are several knobs available to the LLM inference provider to use for being energy efficient. We characterize the impact of these knobs on the latency, throughput, as well as the energy. By exploring these trade-offs, we offer valuable insights into optimizing energy usage without compromising on performance, thereby paving the way for sustainable and cost-effective LLM deployment in data center environments. |
| title | Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference |
| topic | Artificial Intelligence Hardware Architecture Distributed, Parallel, and Cluster Computing C.0; I.2 |
| url | https://arxiv.org/abs/2403.20306 |