Distilling to Hybrid Attention Models via KL-Guided Layer Selection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909974824747008 |
|---|---|
| author | Li, Yanhong Yang, Songlin Tan, Shawn Mishra, Mayank Panda, Rameswar Zhou, Jiawei Kim, Yoon |
| author_facet | Li, Yanhong Yang, Songlin Tan, Shawn Mishra, Mayank Panda, Rameswar Zhou, Jiawei Kim, Yoon |
| contents | Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improving the inference efficiency of LLMs without requiring expensive pretraining from scratch. A critical factor in the conversion process is layer selection, i.e., deciding on which layers to convert to linear attention variants. This paper describes a simple and efficient recipe for layer selection that uses layer importance scores derived from a small amount of training on generic text data. Once the layers have been selected we use a recent pipeline for the distillation process itself \citep[RADLADS;][]{goldstein2025radlads}, which consists of attention weight transfer, hidden state alignment, KL-based distribution matching, followed by a small amount of finetuning. We find that this approach is more effective than existing approaches for layer selection, including heuristics that uniformly interleave linear attentions based on a fixed ratio, as well as more involved approaches that rely on specialized diagnostic datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_20569 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distilling to Hybrid Attention Models via KL-Guided Layer Selection Li, Yanhong Yang, Songlin Tan, Shawn Mishra, Mayank Panda, Rameswar Zhou, Jiawei Kim, Yoon Computation and Language Artificial Intelligence Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improving the inference efficiency of LLMs without requiring expensive pretraining from scratch. A critical factor in the conversion process is layer selection, i.e., deciding on which layers to convert to linear attention variants. This paper describes a simple and efficient recipe for layer selection that uses layer importance scores derived from a small amount of training on generic text data. Once the layers have been selected we use a recent pipeline for the distillation process itself \citep[RADLADS;][]{goldstein2025radlads}, which consists of attention weight transfer, hidden state alignment, KL-based distribution matching, followed by a small amount of finetuning. We find that this approach is more effective than existing approaches for layer selection, including heuristics that uniformly interleave linear attentions based on a fixed ratio, as well as more involved approaches that rely on specialized diagnostic datasets. |
| title | Distilling to Hybrid Attention Models via KL-Guided Layer Selection |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.20569 |