Analyzing the Generalization and Reliability of Steering Vectors
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913817856835584 |
|---|---|
| author | Tan, Daniel Chanin, David Lynch, Aengus Kanoulas, Dimitrios Paige, Brooks Garriga-Alonso, Adria Kirk, Robert |
| author_facet | Tan, Daniel Chanin, David Lynch, Aengus Kanoulas, Dimitrios Paige, Brooks Garriga-Alonso, Adria Kirk, Robert |
| contents | Steering vectors (SVs) have been proposed as an effective approach to adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Overall, our findings show that while steering can work well in the right circumstances, there remain technical difficulties of applying steering vectors to guide models' behaviour at scale. Our code is available at https://github.com/dtch1997/steering-bench |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12404 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Analyzing the Generalization and Reliability of Steering Vectors Tan, Daniel Chanin, David Lynch, Aengus Kanoulas, Dimitrios Paige, Brooks Garriga-Alonso, Adria Kirk, Robert Machine Learning Steering vectors (SVs) have been proposed as an effective approach to adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Overall, our findings show that while steering can work well in the right circumstances, there remain technical difficulties of applying steering vectors to guide models' behaviour at scale. Our code is available at https://github.com/dtch1997/steering-bench |
| title | Analyzing the Generalization and Reliability of Steering Vectors |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.12404 |