Token Level Routing Inference System for Edge Devices
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908311922671616 |
|---|---|
| author | She, Jianshu Zheng, Wenhao Liu, Zhengzhong Wang, Hongyi Xing, Eric Yao, Huaxiu Ho, Qirong |
| author_facet | She, Jianshu Zheng, Wenhao Liu, Zhengzhong Wang, Hongyi Xing, Eric Yao, Huaxiu Ho, Qirong |
| contents | The computational complexity of large language model (LLM) inference significantly constrains their deployment efficiency on edge devices. In contrast, small language models offer faster decoding and lower resource consumption but often suffer from degraded response quality and heightened susceptibility to hallucinations. To address this trade-off, collaborative decoding, in which a large model assists in generating critical tokens, has emerged as a promising solution. This paradigm leverages the strengths of both model types by enabling high-quality inference through selective intervention of the large model, while maintaining the speed and efficiency of the smaller model. In this work, we present a novel collaborative decoding inference system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. Remarkably, the system achieves a 60% performance gain on CommonsenseQA using only a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07878 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Token Level Routing Inference System for Edge Devices She, Jianshu Zheng, Wenhao Liu, Zhengzhong Wang, Hongyi Xing, Eric Yao, Huaxiu Ho, Qirong Computation and Language Distributed, Parallel, and Cluster Computing The computational complexity of large language model (LLM) inference significantly constrains their deployment efficiency on edge devices. In contrast, small language models offer faster decoding and lower resource consumption but often suffer from degraded response quality and heightened susceptibility to hallucinations. To address this trade-off, collaborative decoding, in which a large model assists in generating critical tokens, has emerged as a promising solution. This paradigm leverages the strengths of both model types by enabling high-quality inference through selective intervention of the large model, while maintaining the speed and efficiency of the smaller model. In this work, we present a novel collaborative decoding inference system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. Remarkably, the system achieves a 60% performance gain on CommonsenseQA using only a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud. |
| title | Token Level Routing Inference System for Edge Devices |
| topic | Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2504.07878 |