Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.18397 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912726168633344 |
|---|---|
| author | MacDiarmid, Monte Wright, Benjamin Uesato, Jonathan Benton, Joe Kutasov, Jon Price, Sara Bouscal, Naia Bowman, Sam Bricken, Trenton Cloud, Alex Denison, Carson Gasteiger, Johannes Greenblatt, Ryan Leike, Jan Lindsey, Jack Mikulik, Vlad Perez, Ethan Rodrigues, Alex Thomas, Drake Webson, Albert Ziegler, Daniel Hubinger, Evan |
| author_facet | MacDiarmid, Monte Wright, Benjamin Uesato, Jonathan Benton, Joe Kutasov, Jon Price, Sara Bouscal, Naia Bowman, Sam Bricken, Trenton Cloud, Alex Denison, Carson Gasteiger, Johannes Greenblatt, Ryan Leike, Jan Lindsey, Jack Mikulik, Vlad Perez, Ethan Rodrigues, Alex Thomas, Drake Webson, Albert Ziegler, Daniel Hubinger, Evan |
| contents | We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_18397 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Natural Emergent Misalignment from Reward Hacking in Production RL MacDiarmid, Monte Wright, Benjamin Uesato, Jonathan Benton, Joe Kutasov, Jon Price, Sara Bouscal, Naia Bowman, Sam Bricken, Trenton Cloud, Alex Denison, Carson Gasteiger, Johannes Greenblatt, Ryan Leike, Jan Lindsey, Jack Mikulik, Vlad Perez, Ethan Rodrigues, Alex Thomas, Drake Webson, Albert Ziegler, Daniel Hubinger, Evan Artificial Intelligence Software Engineering We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned. |
| title | Natural Emergent Misalignment from Reward Hacking in Production RL |
| topic | Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2511.18397 |