ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908412397223936 |
|---|---|
| author | Chen, Hao Li, Haoze Xiao, Zhiqing Gao, Lirong Zhang, Qi Hu, Xiaomeng Wang, Ningtao Fu, Xing Zhao, Junbo |
| author_facet | Chen, Hao Li, Haoze Xiao, Zhiqing Gao, Lirong Zhang, Qi Hu, Xiaomeng Wang, Ningtao Fu, Xing Zhao, Junbo |
| contents | Aligning general-purpose large language models (LLMs) to downstream tasks often incurs significant training adjustment costs. Prior research has explored various avenues to enhance alignment efficiency, primarily through minimal-data training or data-driven activations to identify key attention heads. However, these approaches inherently introduce data dependency, which hinders generalization and reusability. To address this issue and enhance model alignment efficiency, we propose the Attention Localization and Pruning Strategy (ALPS), an efficient algorithm that localizes the most task-sensitive attention heads and prunes by restricting attention training updates to these heads, thereby reducing alignment costs. Experimental results demonstrate that our method activates only 10% of attention parameters during fine-tuning while achieving a 2% performance improvement over baselines on three tasks. Moreover, the identified task-specific heads are transferable across datasets and mitigate knowledge forgetting. Our work and findings provide a novel perspective on efficient LLM alignment. The code is available at https://github.com/VoiceBeer/ALPS. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18799 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models Chen, Hao Li, Haoze Xiao, Zhiqing Gao, Lirong Zhang, Qi Hu, Xiaomeng Wang, Ningtao Fu, Xing Zhao, Junbo Computation and Language Artificial Intelligence Aligning general-purpose large language models (LLMs) to downstream tasks often incurs significant training adjustment costs. Prior research has explored various avenues to enhance alignment efficiency, primarily through minimal-data training or data-driven activations to identify key attention heads. However, these approaches inherently introduce data dependency, which hinders generalization and reusability. To address this issue and enhance model alignment efficiency, we propose the Attention Localization and Pruning Strategy (ALPS), an efficient algorithm that localizes the most task-sensitive attention heads and prunes by restricting attention training updates to these heads, thereby reducing alignment costs. Experimental results demonstrate that our method activates only 10% of attention parameters during fine-tuning while achieving a 2% performance improvement over baselines on three tasks. Moreover, the identified task-specific heads are transferable across datasets and mitigate knowledge forgetting. Our work and findings provide a novel perspective on efficient LLM alignment. The code is available at https://github.com/VoiceBeer/ALPS. |
| title | ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.18799 |