Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915913231499264 |
|---|---|
| author | Liu, Zihe Mao, Yulong Xu, Jinan Peng, Xinrui Huang, Kaiyu |
| author_facet | Liu, Zihe Mao, Yulong Xu, Jinan Peng, Xinrui Huang, Kaiyu |
| contents | Knowledge distillation is an effective technique for pre-trained language model compression. However, existing methods only focus on the knowledge distribution among layers, which may cause the loss of fine-grained information in the alignment process. To address this issue, we introduce the Multi-aspect Knowledge Distillation (MaKD) method, which mimics the self-attention and feed-forward modules in greater depth to capture rich language knowledge information at different aspects. Experimental results demonstrate that MaKD can achieve competitive performance compared with various strong baselines with the same storage parameter budget. In addition, our method also performs well in distilling auto-regressive architecture models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_03110 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization Liu, Zihe Mao, Yulong Xu, Jinan Peng, Xinrui Huang, Kaiyu Computation and Language Knowledge distillation is an effective technique for pre-trained language model compression. However, existing methods only focus on the knowledge distribution among layers, which may cause the loss of fine-grained information in the alignment process. To address this issue, we introduce the Multi-aspect Knowledge Distillation (MaKD) method, which mimics the self-attention and feed-forward modules in greater depth to capture rich language knowledge information at different aspects. Experimental results demonstrate that MaKD can achieve competitive performance compared with various strong baselines with the same storage parameter budget. In addition, our method also performs well in distilling auto-regressive architecture models. |
| title | Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.03110 |