Task-driven Layerwise Additive Activation Intervention
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915144531968000 |
|---|---|
| author | Nguyen, Hieu Trung Nguyen, Bao Nguyen, Binh Nguyen, Viet Anh |
| author_facet | Nguyen, Hieu Trung Nguyen, Bao Nguyen, Binh Nguyen, Viet Anh |
| contents | Modern language models (LMs) have significantly advanced generative modeling in natural language processing (NLP). Despite their success, LMs often struggle with adaptation to new contexts in real-time applications. A promising approach to task adaptation is activation intervention, which steers the LMs' generation process by identifying and manipulating the activations. However, existing interventions are highly dependent on heuristic rules or require many prompt inputs to determine effective interventions. This paper proposes a layer-wise additive activation intervention framework that optimizes the intervention process, thus enhancing the sample efficiency. We benchmark our framework on various datasets, demonstrating improvements in the accuracy of pre-trained LMs and competing intervention baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06115 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Task-driven Layerwise Additive Activation Intervention Nguyen, Hieu Trung Nguyen, Bao Nguyen, Binh Nguyen, Viet Anh Computation and Language Machine Learning Modern language models (LMs) have significantly advanced generative modeling in natural language processing (NLP). Despite their success, LMs often struggle with adaptation to new contexts in real-time applications. A promising approach to task adaptation is activation intervention, which steers the LMs' generation process by identifying and manipulating the activations. However, existing interventions are highly dependent on heuristic rules or require many prompt inputs to determine effective interventions. This paper proposes a layer-wise additive activation intervention framework that optimizes the intervention process, thus enhancing the sample efficiency. We benchmark our framework on various datasets, demonstrating improvements in the accuracy of pre-trained LMs and competing intervention baselines. |
| title | Task-driven Layerwise Additive Activation Intervention |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.06115 |