Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.05246 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913930444537856 |
|---|---|
| author | Emmons, Scott Jenner, Erik Elson, David K. Saurous, Rif A. Rajamanoharan, Senthooran Chen, Heng Shafkat, Irhum Shah, Rohin |
| author_facet | Emmons, Scott Jenner, Erik Elson, David K. Saurous, Rif A. Rajamanoharan, Senthooran Chen, Heng Shafkat, Irhum Shah, Rohin |
| contents | While chain-of-thought (CoT) monitoring is an appealing AI safety defense, recent work on "unfaithfulness" has cast doubt on its reliability. These findings highlight an important failure mode, particularly when CoT acts as a post-hoc rationalization in applications like auditing for bias. However, for the distinct problem of runtime monitoring to prevent severe harm, we argue the key property is not faithfulness but monitorability. To this end, we introduce a conceptual framework distinguishing CoT-as-rationalization from CoT-as-computation. We expect that certain classes of severe harm will require complex, multi-step reasoning that necessitates CoT-as-computation. Replicating the experimental setups of prior work, we increase the difficulty of the bad behavior to enforce this necessity condition; this forces the model to expose its reasoning, making it monitorable. We then present methodology guidelines to stress-test CoT monitoring against deliberate evasion. Applying these guidelines, we find that models can learn to obscure their intentions, but only when given significant help, such as detailed human-written strategies or iterative optimization against the monitor. We conclude that, while not infallible, CoT monitoring offers a substantial layer of defense that requires active protection and continued stress-testing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_05246 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors Emmons, Scott Jenner, Erik Elson, David K. Saurous, Rif A. Rajamanoharan, Senthooran Chen, Heng Shafkat, Irhum Shah, Rohin Artificial Intelligence Computation and Language While chain-of-thought (CoT) monitoring is an appealing AI safety defense, recent work on "unfaithfulness" has cast doubt on its reliability. These findings highlight an important failure mode, particularly when CoT acts as a post-hoc rationalization in applications like auditing for bias. However, for the distinct problem of runtime monitoring to prevent severe harm, we argue the key property is not faithfulness but monitorability. To this end, we introduce a conceptual framework distinguishing CoT-as-rationalization from CoT-as-computation. We expect that certain classes of severe harm will require complex, multi-step reasoning that necessitates CoT-as-computation. Replicating the experimental setups of prior work, we increase the difficulty of the bad behavior to enforce this necessity condition; this forces the model to expose its reasoning, making it monitorable. We then present methodology guidelines to stress-test CoT monitoring against deliberate evasion. Applying these guidelines, we find that models can learn to obscure their intentions, but only when given significant help, such as detailed human-written strategies or iterative optimization against the monitor. We conclude that, while not infallible, CoT monitoring offers a substantial layer of defense that requires active protection and continued stress-testing. |
| title | When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2507.05246 |