Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912973200556032 |
|---|---|
| author | Ding, Xuan Sun, Rui Zhang, Yunjian Yan, Xiu Zhou, Yueqi Huang, Kaihao Fu, Suzhong Aviles-Rivero, Angelica I Xie, Chuanlong Zhu, Yao |
| author_facet | Ding, Xuan Sun, Rui Zhang, Yunjian Yan, Xiu Zhou, Yueqi Huang, Kaihao Fu, Suzhong Aviles-Rivero, Angelica I Xie, Chuanlong Zhu, Yao |
| contents | Depth-wise pruning accelerates LLM inference in resource-constrained scenarios but suffers from performance degradation due to direct removal of entire Transformer layers. This paper reveals ``Patch-like'' redundancy across layers via correlation analysis of the outputs of different layers in reproducing kernel Hilbert space, demonstrating consecutive layers exhibit high functional similarity. Building on this observation, this paper proposes Sliding-Window Merging (SWM) - a dynamic compression method that selects consecutive layers from top to bottom using a pre-defined similarity threshold, and compacts patch-redundant layers through a parameter consolidation, thereby simplifying the model structure while maintaining its performance. Extensive experiments on LLMs with various architectures and different parameter scales show that our method outperforms existing pruning techniques in both zero-shot inference performance and retraining recovery quality after pruning. In particular, in the experiment with 35% pruning on the Vicuna-7B model, our method achieved a 1.654% improvement in average performance on zero-shot tasks compared to the existing method. Moreover, we further reveal the potential of combining depth pruning with width pruning to enhance the pruning effect. Our codes are available at https://github.com/920927/SLM-a-sliding-layer-merging-method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19159 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs Ding, Xuan Sun, Rui Zhang, Yunjian Yan, Xiu Zhou, Yueqi Huang, Kaihao Fu, Suzhong Aviles-Rivero, Angelica I Xie, Chuanlong Zhu, Yao Computer Vision and Pattern Recognition Depth-wise pruning accelerates LLM inference in resource-constrained scenarios but suffers from performance degradation due to direct removal of entire Transformer layers. This paper reveals ``Patch-like'' redundancy across layers via correlation analysis of the outputs of different layers in reproducing kernel Hilbert space, demonstrating consecutive layers exhibit high functional similarity. Building on this observation, this paper proposes Sliding-Window Merging (SWM) - a dynamic compression method that selects consecutive layers from top to bottom using a pre-defined similarity threshold, and compacts patch-redundant layers through a parameter consolidation, thereby simplifying the model structure while maintaining its performance. Extensive experiments on LLMs with various architectures and different parameter scales show that our method outperforms existing pruning techniques in both zero-shot inference performance and retraining recovery quality after pruning. In particular, in the experiment with 35% pruning on the Vicuna-7B model, our method achieved a 1.654% improvement in average performance on zero-shot tasks compared to the existing method. Moreover, we further reveal the potential of combining depth pruning with width pruning to enhance the pruning effect. Our codes are available at https://github.com/920927/SLM-a-sliding-layer-merging-method. |
| title | Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.19159 |