Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.09173 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909735254491136 |
|---|---|
| author | Xu, Hao Peng, Long Song, Shezheng Liu, Xiaodong Jun, Ma Li, Shasha Yu, Jie Mao, Xiaoguang |
| author_facet | Xu, Hao Peng, Long Song, Shezheng Liu, Xiaodong Jun, Ma Li, Shasha Yu, Jie Mao, Xiaoguang |
| contents | Most Large Language Models (LLMs) are currently deployed in the cloud, with users relying on internet connectivity for access. However, this paradigm faces challenges such as network latency, privacy concerns, and bandwidth limits. Thus, deploying LLMs on edge devices has become an important research focus. In edge inference, request latency is critical as high latency can impair real-time tasks. At the same time, edge devices usually have limited battery capacity, making energy consumption another major concern. Balancing energy consumption and inference latency is essential. To address this, we propose an LLM inference energy management framework that optimizes GPU frequency and batch size to balance latency and energy consumption. By effectively managing the exploration-exploitation dilemma in configuration search, the framework finds the optimal settings. The framework was implemented on the NVIDIA Jetson AGX Orin platform, and a series of experimental validations were conducted. Results demonstrate that, compared to the default configuration, our framework reduces energy delay product (EDP) by 12.4%-29.9%, achieving a better balance between energy consumption and latency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09173 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Camel: Energy-Aware LLM Inference on Resource-Constrained Devices Xu, Hao Peng, Long Song, Shezheng Liu, Xiaodong Jun, Ma Li, Shasha Yu, Jie Mao, Xiaoguang Networking and Internet Architecture Most Large Language Models (LLMs) are currently deployed in the cloud, with users relying on internet connectivity for access. However, this paradigm faces challenges such as network latency, privacy concerns, and bandwidth limits. Thus, deploying LLMs on edge devices has become an important research focus. In edge inference, request latency is critical as high latency can impair real-time tasks. At the same time, edge devices usually have limited battery capacity, making energy consumption another major concern. Balancing energy consumption and inference latency is essential. To address this, we propose an LLM inference energy management framework that optimizes GPU frequency and batch size to balance latency and energy consumption. By effectively managing the exploration-exploitation dilemma in configuration search, the framework finds the optimal settings. The framework was implemented on the NVIDIA Jetson AGX Orin platform, and a series of experimental validations were conducted. Results demonstrate that, compared to the default configuration, our framework reduces energy delay product (EDP) by 12.4%-29.9%, achieving a better balance between energy consumption and latency. |
| title | Camel: Energy-Aware LLM Inference on Resource-Constrained Devices |
| topic | Networking and Internet Architecture |
| url | https://arxiv.org/abs/2508.09173 |