Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2605.10664 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910218001055744 |
|---|---|
| author | Kang, Diancheng Liu, Zheyuan Ma, Ningshan Huang, Yue Tan, Zhaoxuan Jiang, Meng |
| author_facet | Kang, Diancheng Liu, Zheyuan Ma, Ningshan Huang, Yue Tan, Zhaoxuan Jiang, Meng |
| contents | Activation steering controls language model behavior by adding directions to internal representations at inference time, but standard residual-stream steering can fail in stateful dialogue. We identify KV-cache contamination as a key failure mode: steered token states are stored and repeatedly reused, turning a local perturbation into cumulative coherence degradation. To address this challenge, we propose Gated Cropped Attention-Delta steering (GCAD), which extracts steering signals from system-prompt contributions to self-attention and applies them with token-level gating. Across persona-steering experiments, GCAD preserves trait control while substantially improving long-horizon coherence. On the main multi-turn benchmark, GCAD improves average coherence drift from -18.6 to -1.9 and raises turn-10 trait expression from 78.0 to 93.1. These results suggest that activation steering becomes more reliable when interventions follow the prompt-mediated pathways that models already use for behavioral control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10664 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions Kang, Diancheng Liu, Zheyuan Ma, Ningshan Huang, Yue Tan, Zhaoxuan Jiang, Meng Computation and Language Artificial Intelligence Activation steering controls language model behavior by adding directions to internal representations at inference time, but standard residual-stream steering can fail in stateful dialogue. We identify KV-cache contamination as a key failure mode: steered token states are stored and repeatedly reused, turning a local perturbation into cumulative coherence degradation. To address this challenge, we propose Gated Cropped Attention-Delta steering (GCAD), which extracts steering signals from system-prompt contributions to self-attention and applies them with token-level gating. Across persona-steering experiments, GCAD preserves trait control while substantially improving long-horizon coherence. On the main multi-turn benchmark, GCAD improves average coherence drift from -18.6 to -1.9 and raises turn-10 trait expression from 78.0 to 93.1. These results suggest that activation steering becomes more reliable when interventions follow the prompt-mediated pathways that models already use for behavioral control. |
| title | Prompt-Activation Duality: Improving Activation Steering via Attention-Level Interventions |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.10664 |