IteRABRe: Iterative Recovery-Aided Block Reduction
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910865234591744 |
|---|---|
| author | Wibowo, Haryo Akbarianto Song, Haiyue Tanaka, Hideki Utiyama, Masao Aji, Alham Fikri Dabre, Raj |
| author_facet | Wibowo, Haryo Akbarianto Song, Haiyue Tanaka, Hideki Utiyama, Masao Aji, Alham Fikri Dabre, Raj |
| contents | Large Language Models (LLMs) have grown increasingly expensive to deploy, driving the need for effective model compression techniques. While block pruning offers a straightforward approach to reducing model size, existing methods often struggle to maintain performance or require substantial computational resources for recovery. We present IteRABRe, a simple yet effective iterative pruning method that achieves superior compression results while requiring minimal computational resources. Using only 2.5M tokens for recovery, our method outperforms baseline approaches by ~3% on average when compressing the Llama3.1-8B and Qwen2.5-7B models. IteRABRe demonstrates particular strength in the preservation of linguistic capabilities, showing an improvement 5% over the baselines in language-related tasks. Our analysis reveals distinct pruning characteristics between these models, while also demonstrating preservation of multilingual capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06291 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IteRABRe: Iterative Recovery-Aided Block Reduction Wibowo, Haryo Akbarianto Song, Haiyue Tanaka, Hideki Utiyama, Masao Aji, Alham Fikri Dabre, Raj Computation and Language 68T50 Large Language Models (LLMs) have grown increasingly expensive to deploy, driving the need for effective model compression techniques. While block pruning offers a straightforward approach to reducing model size, existing methods often struggle to maintain performance or require substantial computational resources for recovery. We present IteRABRe, a simple yet effective iterative pruning method that achieves superior compression results while requiring minimal computational resources. Using only 2.5M tokens for recovery, our method outperforms baseline approaches by ~3% on average when compressing the Llama3.1-8B and Qwen2.5-7B models. IteRABRe demonstrates particular strength in the preservation of linguistic capabilities, showing an improvement 5% over the baselines in language-related tasks. Our analysis reveals distinct pruning characteristics between these models, while also demonstrating preservation of multilingual capabilities. |
| title | IteRABRe: Iterative Recovery-Aided Block Reduction |
| topic | Computation and Language 68T50 |
| url | https://arxiv.org/abs/2503.06291 |