Output Supervision Can Obfuscate the Chain of Thought
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909903144091648 |
|---|---|
| author | Drori, Jacob Marks, Luke Woodworth, Bryce Cloud, Alex Turner, Alexander Matt |
| author_facet | Drori, Jacob Marks, Luke Woodworth, Bryce Cloud, Alex Turner, Alexander Matt |
| contents | OpenAI (2025) showed that training against a chain of thought (CoT) monitor can cause obfuscated CoTs, which contain bad behavior the monitor cannot detect. They proposed to keep CoTs monitorable by training only against output monitors that do not have access to CoT. We show that such training can still cause obfuscated CoTs via two mechanisms. First, when a model is trained to produce a safe-looking output, that model may generalize to making its CoTs look safe. Second, since later tokens are conditioned on earlier ones, safe-looking CoTs may increase the likelihood of safe outputs, causing safe-looking CoTs to be reinforced. We introduce two mitigations to address these two issues, which achieve a Pareto improvement in terms of monitorability and task performance compared to regular training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11584 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Output Supervision Can Obfuscate the Chain of Thought Drori, Jacob Marks, Luke Woodworth, Bryce Cloud, Alex Turner, Alexander Matt Machine Learning Artificial Intelligence Cryptography and Security OpenAI (2025) showed that training against a chain of thought (CoT) monitor can cause obfuscated CoTs, which contain bad behavior the monitor cannot detect. They proposed to keep CoTs monitorable by training only against output monitors that do not have access to CoT. We show that such training can still cause obfuscated CoTs via two mechanisms. First, when a model is trained to produce a safe-looking output, that model may generalize to making its CoTs look safe. Second, since later tokens are conditioned on earlier ones, safe-looking CoTs may increase the likelihood of safe outputs, causing safe-looking CoTs to be reinforced. We introduce two mitigations to address these two issues, which achieve a Pareto improvement in terms of monitorability and task performance compared to regular training. |
| title | Output Supervision Can Obfuscate the Chain of Thought |
| topic | Machine Learning Artificial Intelligence Cryptography and Security |
| url | https://arxiv.org/abs/2511.11584 |