Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.16251 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913450012180480 |
|---|---|
| author | Agarwal, Divyansh Fabbri, Alexander R. Risher, Ben Laban, Philippe Joty, Shafiq Wu, Chien-Sheng |
| author_facet | Agarwal, Divyansh Fabbri, Alexander R. Risher, Ben Laban, Philippe Joty, Shafiq Wu, Chien-Sheng |
| contents | Prompt leakage poses a compelling security and privacy threat in LLM applications. Leakage of system prompts may compromise intellectual property, and act as adversarial reconnaissance for an attacker. A systematic evaluation of prompt leakage threats and mitigation strategies is lacking, especially for multi-turn LLM interactions. In this paper, we systematically investigate LLM vulnerabilities against prompt leakage for 10 closed- and open-source LLMs, across four domains. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. Our standardized setup further allows dissecting leakage of specific prompt contents such as task instructions and knowledge documents. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts. We present different combination of defenses against our threat model, including a cost analysis. Our study highlights key takeaways for building secure LLM applications and provides directions for research in multi-turn LLM interactions |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16251 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Prompt Leakage effect and defense strategies for multi-turn LLM interactions Agarwal, Divyansh Fabbri, Alexander R. Risher, Ben Laban, Philippe Joty, Shafiq Wu, Chien-Sheng Cryptography and Security Artificial Intelligence Computation and Language Prompt leakage poses a compelling security and privacy threat in LLM applications. Leakage of system prompts may compromise intellectual property, and act as adversarial reconnaissance for an attacker. A systematic evaluation of prompt leakage threats and mitigation strategies is lacking, especially for multi-turn LLM interactions. In this paper, we systematically investigate LLM vulnerabilities against prompt leakage for 10 closed- and open-source LLMs, across four domains. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. Our standardized setup further allows dissecting leakage of specific prompt contents such as task instructions and knowledge documents. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts. We present different combination of defenses against our threat model, including a cost analysis. Our study highlights key takeaways for building secure LLM applications and provides directions for research in multi-turn LLM interactions |
| title | Prompt Leakage effect and defense strategies for multi-turn LLM interactions |
| topic | Cryptography and Security Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2404.16251 |