Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10965 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916664415617024 |
|---|---|
| author | Marks, Samuel Treutlein, Johannes Bricken, Trenton Lindsey, Jack Marcus, Jonathan Mishra-Sharma, Siddharth Ziegler, Daniel Ameisen, Emmanuel Batson, Joshua Belonax, Tim Bowman, Samuel R. Carter, Shan Chen, Brian Cunningham, Hoagy Denison, Carson Dietz, Florian Golechha, Satvik Khan, Akbir Kirchner, Jan Leike, Jan Meek, Austin Nishimura-Gasparian, Kei Ong, Euan Olah, Christopher Pearce, Adam Roger, Fabien Salle, Jeanne Shih, Andy Tong, Meg Thomas, Drake Rivoire, Kelley Jermyn, Adam MacDiarmid, Monte Henighan, Tom Hubinger, Evan |
| author_facet | Marks, Samuel Treutlein, Johannes Bricken, Trenton Lindsey, Jack Marcus, Jonathan Mishra-Sharma, Siddharth Ziegler, Daniel Ameisen, Emmanuel Batson, Joshua Belonax, Tim Bowman, Samuel R. Carter, Shan Chen, Brian Cunningham, Hoagy Denison, Carson Dietz, Florian Golechha, Satvik Khan, Akbir Kirchner, Jan Leike, Jan Meek, Austin Nishimura-Gasparian, Kei Ong, Euan Olah, Christopher Pearce, Adam Roger, Fabien Salle, Jeanne Shih, Andy Tong, Meg Thomas, Drake Rivoire, Kelley Jermyn, Adam MacDiarmid, Monte Henighan, Tom Hubinger, Evan |
| contents | We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objective. Our training pipeline first teaches the model about exploitable errors in RLHF reward models (RMs), then trains the model to exploit some of these errors. We verify via out-of-distribution evaluations that the model generalizes to exhibit whatever behaviors it believes RMs rate highly, including ones not reinforced during training. We leverage this model to study alignment audits in two ways. First, we conduct a blind auditing game where four teams, unaware of the model's hidden objective or training, investigate it for concerning behaviors and their causes. Three teams successfully uncovered the model's hidden objective using techniques including interpretability with sparse autoencoders (SAEs), behavioral attacks, and training data analysis. Second, we conduct an unblinded follow-up study of eight techniques for auditing the model, analyzing their strengths and limitations. Overall, our work provides a concrete example of using alignment audits to discover a model's hidden objective and proposes a methodology for practicing and validating progress in alignment auditing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10965 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Auditing language models for hidden objectives Marks, Samuel Treutlein, Johannes Bricken, Trenton Lindsey, Jack Marcus, Jonathan Mishra-Sharma, Siddharth Ziegler, Daniel Ameisen, Emmanuel Batson, Joshua Belonax, Tim Bowman, Samuel R. Carter, Shan Chen, Brian Cunningham, Hoagy Denison, Carson Dietz, Florian Golechha, Satvik Khan, Akbir Kirchner, Jan Leike, Jan Meek, Austin Nishimura-Gasparian, Kei Ong, Euan Olah, Christopher Pearce, Adam Roger, Fabien Salle, Jeanne Shih, Andy Tong, Meg Thomas, Drake Rivoire, Kelley Jermyn, Adam MacDiarmid, Monte Henighan, Tom Hubinger, Evan Artificial Intelligence Computation and Language Machine Learning We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objective. Our training pipeline first teaches the model about exploitable errors in RLHF reward models (RMs), then trains the model to exploit some of these errors. We verify via out-of-distribution evaluations that the model generalizes to exhibit whatever behaviors it believes RMs rate highly, including ones not reinforced during training. We leverage this model to study alignment audits in two ways. First, we conduct a blind auditing game where four teams, unaware of the model's hidden objective or training, investigate it for concerning behaviors and their causes. Three teams successfully uncovered the model's hidden objective using techniques including interpretability with sparse autoencoders (SAEs), behavioral attacks, and training data analysis. Second, we conduct an unblinded follow-up study of eight techniques for auditing the model, analyzing their strengths and limitations. Overall, our work provides a concrete example of using alignment audits to discover a model's hidden objective and proposes a methodology for practicing and validating progress in alignment auditing. |
| title | Auditing language models for hidden objectives |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.10965 |