The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910415605202944 |
|---|---|
| author | Wallace, Eric Xiao, Kai Leike, Reimar Weng, Lilian Heidecke, Johannes Beutel, Alex |
| author_facet | Wallace, Eric Xiao, Kai Leike, Reimar Weng, Lilian Heidecke, Johannes Beutel, Alex |
| contents | Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_13208 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions Wallace, Eric Xiao, Kai Leike, Reimar Weng, Lilian Heidecke, Johannes Beutel, Alex Cryptography and Security Computation and Language Machine Learning Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities. |
| title | The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions |
| topic | Cryptography and Security Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.13208 |