Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917453530923008 |
|---|---|
| author | Lucas, Ryan Behdin, Kayhan Wang, Zhipeng Song, Qingquan Tang, Shao Mazumder, Rahul |
| author_facet | Lucas, Ryan Behdin, Kayhan Wang, Zhipeng Song, Qingquan Tang, Shao Mazumder, Rahul |
| contents | Reasoning language models such as DeepSeek-R1 produce long chain-of-thought traces during inference time which make them costly to deploy at scale. We show that using compression techniques such as neural network pruning produces greater performance loss than in typical language modeling tasks, and in some cases can make the model slower since they cause the model to produce more thinking tokens but with worse performance. We show that this is partly due to the fact that standard LLM pruning methods often focus on input reconstruction, whereas reasoning is a decode-dominated task. We introduce a simple, drop-in fix: during pruning we jointly reconstruct activations from the input and the model's on-policy chain-of-thought traces. This "Reasoning-Aware Compression" (RAC) integrates seamlessly into existing pruning workflows such as SparseGPT, and boosts their performance significantly. Code reproducing the results in the paper can be found at: https://github.com/RyanLucas3/RAC |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12464 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction Lucas, Ryan Behdin, Kayhan Wang, Zhipeng Song, Qingquan Tang, Shao Mazumder, Rahul Artificial Intelligence Reasoning language models such as DeepSeek-R1 produce long chain-of-thought traces during inference time which make them costly to deploy at scale. We show that using compression techniques such as neural network pruning produces greater performance loss than in typical language modeling tasks, and in some cases can make the model slower since they cause the model to produce more thinking tokens but with worse performance. We show that this is partly due to the fact that standard LLM pruning methods often focus on input reconstruction, whereas reasoning is a decode-dominated task. We introduce a simple, drop-in fix: during pruning we jointly reconstruct activations from the input and the model's on-policy chain-of-thought traces. This "Reasoning-Aware Compression" (RAC) integrates seamlessly into existing pruning workflows such as SparseGPT, and boosts their performance significantly. Code reproducing the results in the paper can be found at: https://github.com/RyanLucas3/RAC |
| title | Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.12464 |