Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908360137244672 |
|---|---|
| author | Yan, Guang Zhang, Yuhui Guo, Zimu Zhao, Lutan Chen, Xiaojun Wang, Chen Wang, Wenhao Meng, Dan Hou, Rui |
| author_facet | Yan, Guang Zhang, Yuhui Guo, Zimu Zhao, Lutan Chen, Xiaojun Wang, Chen Wang, Wenhao Meng, Dan Hou, Rui |
| contents | With the growing use of large language models (LLMs) hosted on cloud platforms to offer inference services, privacy concerns about the potential leakage of sensitive information are escalating. Secure multi-party computation (MPC) is a promising solution to protect the privacy in LLM inference. However, MPC requires frequent inter-server communication, causing high performance overhead.
Inspired by the prevalent activation sparsity of LLMs, where most neuron are not activated after non-linear activation functions, we propose an efficient private inference system, Comet. This system employs an accurate and fast predictor to predict the sparsity distribution of activation function output. Additionally, we introduce a new private inference protocol. It efficiently and securely avoids computations involving zero values by exploiting the spatial locality of the predicted sparse distribution. While this computation-avoidance approach impacts the spatiotemporal continuity of KV cache entries, we address this challenge with a low-communication overhead cache refilling strategy that merges miss requests and incorporates a prefetching mechanism. Finally, we evaluate Comet on four common LLMs and compare it with six state-of-the-art private inference systems. Comet achieves a 1.87x-2.63x speedup and a 1.94x-2.64x communication reduction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07239 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity Yan, Guang Zhang, Yuhui Guo, Zimu Zhao, Lutan Chen, Xiaojun Wang, Chen Wang, Wenhao Meng, Dan Hou, Rui Cryptography and Security Artificial Intelligence With the growing use of large language models (LLMs) hosted on cloud platforms to offer inference services, privacy concerns about the potential leakage of sensitive information are escalating. Secure multi-party computation (MPC) is a promising solution to protect the privacy in LLM inference. However, MPC requires frequent inter-server communication, causing high performance overhead. Inspired by the prevalent activation sparsity of LLMs, where most neuron are not activated after non-linear activation functions, we propose an efficient private inference system, Comet. This system employs an accurate and fast predictor to predict the sparsity distribution of activation function output. Additionally, we introduce a new private inference protocol. It efficiently and securely avoids computations involving zero values by exploiting the spatial locality of the predicted sparse distribution. While this computation-avoidance approach impacts the spatiotemporal continuity of KV cache entries, we address this challenge with a low-communication overhead cache refilling strategy that merges miss requests and incorporates a prefetching mechanism. Finally, we evaluate Comet on four common LLMs and compare it with six state-of-the-art private inference systems. Comet achieves a 1.87x-2.63x speedup and a 1.94x-2.64x communication reduction. |
| title | Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2505.07239 |