Detecting High-Stakes Interactions with Activation Probes
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911392713408512 |
|---|---|
| author | McKenzie, Alex Pawar, Urja Blandfort, Phil Bankes, William Krueger, David Lubana, Ekdeep Singh Krasheninnikov, Dmitrii |
| author_facet | McKenzie, Alex Pawar, Urja Blandfort, Phil Bankes, William Krueger, David Lubana, Ekdeep Singh Krasheninnikov, Dmitrii |
| contents | Monitoring is an important aspect of safely deploying Large Language Models (LLMs). This paper examines activation probes for detecting ``high-stakes'' interactions -- where the text indicates that the interaction might lead to significant harm -- as a critical, yet underexplored, target for such monitoring. We evaluate several probe architectures trained on synthetic data, and find them to exhibit robust generalization to diverse, out-of-distribution, real-world data. Probes' performance is comparable to that of prompted or finetuned medium-sized LLM monitors, while offering computational savings of six orders-of-magnitude. These savings are enabled by reusing activations of the model that is being monitored. Our experiments also highlight the potential of building resource-aware hierarchical monitoring systems, where probes serve as an efficient initial filter and flag cases for more expensive downstream analysis. We release our novel synthetic dataset and the codebase at https://github.com/arrrlex/models-under-pressure. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10805 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Detecting High-Stakes Interactions with Activation Probes McKenzie, Alex Pawar, Urja Blandfort, Phil Bankes, William Krueger, David Lubana, Ekdeep Singh Krasheninnikov, Dmitrii Machine Learning Monitoring is an important aspect of safely deploying Large Language Models (LLMs). This paper examines activation probes for detecting ``high-stakes'' interactions -- where the text indicates that the interaction might lead to significant harm -- as a critical, yet underexplored, target for such monitoring. We evaluate several probe architectures trained on synthetic data, and find them to exhibit robust generalization to diverse, out-of-distribution, real-world data. Probes' performance is comparable to that of prompted or finetuned medium-sized LLM monitors, while offering computational savings of six orders-of-magnitude. These savings are enabled by reusing activations of the model that is being monitored. Our experiments also highlight the potential of building resource-aware hierarchical monitoring systems, where probes serve as an efficient initial filter and flag cases for more expensive downstream analysis. We release our novel synthetic dataset and the codebase at https://github.com/arrrlex/models-under-pressure. |
| title | Detecting High-Stakes Interactions with Activation Probes |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.10805 |