Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910876518318080 |
|---|---|
| author | Poddar, Soham Koley, Paramita Misra, Janardan Podder, Sanjay Ganguly, Niloy Ghosh, Saptarshi |
| author_facet | Poddar, Soham Koley, Paramita Misra, Janardan Podder, Sanjay Ganguly, Niloy Ghosh, Saptarshi |
| contents | Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05610 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models Poddar, Soham Koley, Paramita Misra, Janardan Podder, Sanjay Ganguly, Niloy Ghosh, Saptarshi Computation and Language Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment. |
| title | Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.05610 |