CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908391600816128 |
|---|---|
| author | Tian, Chunlin Qin, Xinpeng Tam, Kahou Li, Li Wang, Zijian Zhao, Yuanzhe Zhang, Minglei Xu, Chengzhong |
| author_facet | Tian, Chunlin Qin, Xinpeng Tam, Kahou Li, Li Wang, Zijian Zhao, Yuanzhe Zhang, Minglei Xu, Chengzhong |
| contents | Deploying large language models (LLMs) on edge devices is crucial for delivering fast responses and ensuring data privacy. However, the limited storage, weight, and power of edge devices make it difficult to deploy LLM-powered applications. These devices must balance latency requirements with energy consumption and model accuracy. In this paper, we first quantify the challenges of deploying LLMs on off-the-shelf edge devices and then we present CLONE, an in-depth algorithm-hardware co-design at both the model- and system-level that intelligently integrates real-time, energy optimization while maintaining robust generality. In order to maximize the synergistic benefits of these algorithms in always-on and intermediate edge computing settings, we specialize in a 28nm scalable hardware accelerator system. We implement and extensively evaluate CLONE on two off-the-shelf edge platforms. Experiments show that CLONE effectively accelerates the inference process up to 11.92x, and saves energy up to 7.36x, while maintaining high-generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02847 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge Tian, Chunlin Qin, Xinpeng Tam, Kahou Li, Li Wang, Zijian Zhao, Yuanzhe Zhang, Minglei Xu, Chengzhong Hardware Architecture Systems and Control Deploying large language models (LLMs) on edge devices is crucial for delivering fast responses and ensuring data privacy. However, the limited storage, weight, and power of edge devices make it difficult to deploy LLM-powered applications. These devices must balance latency requirements with energy consumption and model accuracy. In this paper, we first quantify the challenges of deploying LLMs on off-the-shelf edge devices and then we present CLONE, an in-depth algorithm-hardware co-design at both the model- and system-level that intelligently integrates real-time, energy optimization while maintaining robust generality. In order to maximize the synergistic benefits of these algorithms in always-on and intermediate edge computing settings, we specialize in a 28nm scalable hardware accelerator system. We implement and extensively evaluate CLONE on two off-the-shelf edge platforms. Experiments show that CLONE effectively accelerates the inference process up to 11.92x, and saves energy up to 7.36x, while maintaining high-generation. |
| title | CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge |
| topic | Hardware Architecture Systems and Control |
| url | https://arxiv.org/abs/2506.02847 |