Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.01926 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914179196125184 |
|---|---|
| author | Skaf, Joey Ibanez-Lissen, Luis McCarthy, Robert Watts, Connor Georgiv, Vasil Whittingham, Hannes Gonzalez-Manzano, Lorena Lindner, David Tice, Cameron Young, Edward James Radmard, Puria |
| author_facet | Skaf, Joey Ibanez-Lissen, Luis McCarthy, Robert Watts, Connor Georgiv, Vasil Whittingham, Hannes Gonzalez-Manzano, Lorena Lindner, David Tice, Cameron Young, Edward James Radmard, Puria |
| contents | Chain-of-thought (CoT) reasoning not only enhances large language model performance but also provides critical insights into decision-making processes, marking it as a useful tool for monitoring model intent and planning. However, recent works have shown that banning the mention of a specific example of reward hacking causes obfuscation of the undesired reasoning traces but the persistence of the undesired behavior, threatening the reliability of CoT monitoring. We provide an extension to these results with regard to the ability of models to learn a specific type of obfuscated reasoning: steganography. First, we show that penalizing the use of specific strings within load-bearing reasoning traces causes models to substitute alternative strings. Crucially, this does not alter the underlying method by which the model performs the task, demonstrating that the model can learn to steganographically encode its reasoning.We further demonstrate that models can generalize an encoding scheme. When the penalized strings belong to an overarching class, the model learns not only to substitute strings seen in training, but also develops a general encoding scheme for all members of the class which it can apply to held-out testing strings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01926 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Large language models can learn and generalize steganographic chain-of-thought under process supervision Skaf, Joey Ibanez-Lissen, Luis McCarthy, Robert Watts, Connor Georgiv, Vasil Whittingham, Hannes Gonzalez-Manzano, Lorena Lindner, David Tice, Cameron Young, Edward James Radmard, Puria Artificial Intelligence Computation and Language Machine Learning Chain-of-thought (CoT) reasoning not only enhances large language model performance but also provides critical insights into decision-making processes, marking it as a useful tool for monitoring model intent and planning. However, recent works have shown that banning the mention of a specific example of reward hacking causes obfuscation of the undesired reasoning traces but the persistence of the undesired behavior, threatening the reliability of CoT monitoring. We provide an extension to these results with regard to the ability of models to learn a specific type of obfuscated reasoning: steganography. First, we show that penalizing the use of specific strings within load-bearing reasoning traces causes models to substitute alternative strings. Crucially, this does not alter the underlying method by which the model performs the task, demonstrating that the model can learn to steganographically encode its reasoning.We further demonstrate that models can generalize an encoding scheme. When the penalized strings belong to an overarching class, the model learns not only to substitute strings seen in training, but also develops a general encoding scheme for all members of the class which it can apply to held-out testing strings. |
| title | Large language models can learn and generalize steganographic chain-of-thought under process supervision |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.01926 |