Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.05819 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910196304969728 |
|---|---|
| author | Xu, Shen Zhuge, Xiangwen Xu, Zhe Hu, Yingkun Yang, Zheng Liu, Yunhao |
| author_facet | Xu, Shen Zhuge, Xiangwen Xu, Zhe Hu, Yingkun Yang, Zheng Liu, Yunhao |
| contents | LLMs often struggle with memory-constrained deployment on consumer-grade hardware due to their massive parameter sizes. While existing solutions such as model compression and offloading improve deployment feasibility, they often suffer from substantial accuracy degradation or severe throughput bottlenecks. Recent error compensation methods recover accuracy through auxiliary LoRA-style branches, and we observe that these branches are inherently amenable to offloading: they require substantial parameter storage but access only a small subset of compensation parameters during each inference step. Motivated by this opportunity, we propose HCInfer, a heterogeneous inference system that offloads residual compensation to the CPU while executing the compressed backbone on the GPU, and further introduces an asynchronous compensation pipeline and sensitivity-aware dynamic rank allocation to hide compensation overhead and maximize accuracy recovery. Experimental results show that HCInfer achieves a maximum accuracy improvement of 5.2% on downstream tasks compared to compression model and sustaining a maximum speedup of 10.4x compared to full-precision model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_05819 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices Xu, Shen Zhuge, Xiangwen Xu, Zhe Hu, Yingkun Yang, Zheng Liu, Yunhao Machine Learning LLMs often struggle with memory-constrained deployment on consumer-grade hardware due to their massive parameter sizes. While existing solutions such as model compression and offloading improve deployment feasibility, they often suffer from substantial accuracy degradation or severe throughput bottlenecks. Recent error compensation methods recover accuracy through auxiliary LoRA-style branches, and we observe that these branches are inherently amenable to offloading: they require substantial parameter storage but access only a small subset of compensation parameters during each inference step. Motivated by this opportunity, we propose HCInfer, a heterogeneous inference system that offloads residual compensation to the CPU while executing the compressed backbone on the GPU, and further introduces an asynchronous compensation pipeline and sensitivity-aware dynamic rank allocation to hide compensation overhead and maximize accuracy recovery. Experimental results show that HCInfer achieves a maximum accuracy improvement of 5.2% on downstream tasks compared to compression model and sustaining a maximum speedup of 10.4x compared to full-precision model. |
| title | HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.05819 |