Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929403409203200 |
|---|---|
| author | Denison, Carson MacDiarmid, Monte Barez, Fazl Duvenaud, David Kravec, Shauna Marks, Samuel Schiefer, Nicholas Soklaski, Ryan Tamkin, Alex Kaplan, Jared Shlegeris, Buck Bowman, Samuel R. Perez, Ethan Hubinger, Evan |
| author_facet | Denison, Carson MacDiarmid, Monte Barez, Fazl Duvenaud, David Kravec, Shauna Marks, Samuel Schiefer, Nicholas Soklaski, Ryan Tamkin, Alex Kaplan, Jared Shlegeris, Buck Bowman, Samuel R. Perez, Ethan Hubinger, Evan |
| contents | In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration. In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments. Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering. These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10162 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models Denison, Carson MacDiarmid, Monte Barez, Fazl Duvenaud, David Kravec, Shauna Marks, Samuel Schiefer, Nicholas Soklaski, Ryan Tamkin, Alex Kaplan, Jared Shlegeris, Buck Bowman, Samuel R. Perez, Ethan Hubinger, Evan Artificial Intelligence Computation and Language In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming can range from simple behaviors like sycophancy to sophisticated and pernicious behaviors like reward-tampering, where a model directly modifies its own reward mechanism. However, these more pernicious behaviors may be too complex to be discovered via exploration. In this paper, we study whether Large Language Model (LLM) assistants which find easily discovered forms of specification gaming will generalize to perform rarer and more blatant forms, up to and including reward-tampering. We construct a curriculum of increasingly sophisticated gameable environments and find that training on early-curriculum environments leads to more specification gaming on remaining environments. Strikingly, a small but non-negligible proportion of the time, LLM assistants trained on the full curriculum generalize zero-shot to directly rewriting their own reward function. Retraining an LLM not to game early-curriculum environments mitigates, but does not eliminate, reward-tampering in later environments. Moreover, adding harmlessness training to our gameable environments does not prevent reward-tampering. These results demonstrate that LLMs can generalize from common forms of specification gaming to more pernicious reward tampering and that such behavior may be nontrivial to remove. |
| title | Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.10162 |