When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911438779449344 |
|---|---|
| author | Wei, Jiankun Abdulrazzag, Abdulrahman Zhang, Tianchen Muursepp, Adel Saileshwar, Gururaj |
| author_facet | Wei, Jiankun Abdulrazzag, Abdulrahman Zhang, Tianchen Muursepp, Adel Saileshwar, Gururaj |
| contents | Deployed large language models (LLMs) often rely on speculative decoding, a technique that generates and verifies multiple candidate tokens in parallel, to improve throughput and latency. In this work, we reveal a new side-channel whereby input-dependent patterns of correct and incorrect speculations can be inferred by monitoring per-iteration token counts or packet sizes. In evaluations using research prototypes and production-grade vLLM serving frameworks, we show that an adversary monitoring these patterns can fingerprint user queries (from a set of 50 prompts) with over 75% accuracy across four speculative-decoding schemes at temperature 0.3: REST (100%), LADE (91.6%), BiLD (95.2%), and EAGLE (77.6%). Even at temperature 1.0, accuracy remains far above the 2% random baseline - REST (99.6%), LADE (61.2%), BiLD (63.6%), and EAGLE (24%). We also show the capability of the attacker to leak confidential datastore contents used for prediction at rates exceeding 25 tokens/sec. To defend against these, we propose and evaluate a suite of mitigations, including packet padding and iteration-wise token aggregation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01076 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs Wei, Jiankun Abdulrazzag, Abdulrahman Zhang, Tianchen Muursepp, Adel Saileshwar, Gururaj Computation and Language Artificial Intelligence Cryptography and Security Distributed, Parallel, and Cluster Computing Machine Learning Deployed large language models (LLMs) often rely on speculative decoding, a technique that generates and verifies multiple candidate tokens in parallel, to improve throughput and latency. In this work, we reveal a new side-channel whereby input-dependent patterns of correct and incorrect speculations can be inferred by monitoring per-iteration token counts or packet sizes. In evaluations using research prototypes and production-grade vLLM serving frameworks, we show that an adversary monitoring these patterns can fingerprint user queries (from a set of 50 prompts) with over 75% accuracy across four speculative-decoding schemes at temperature 0.3: REST (100%), LADE (91.6%), BiLD (95.2%), and EAGLE (77.6%). Even at temperature 1.0, accuracy remains far above the 2% random baseline - REST (99.6%), LADE (61.2%), BiLD (63.6%), and EAGLE (24%). We also show the capability of the attacker to leak confidential datastore contents used for prediction at rates exceeding 25 tokens/sec. To defend against these, we propose and evaluate a suite of mitigations, including packet padding and iteration-wise token aggregation. |
| title | When Speculation Spills Secrets: Side Channels via Speculative Decoding In LLMs |
| topic | Computation and Language Artificial Intelligence Cryptography and Security Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2411.01076 |