REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910215056654336 |
|---|---|
| author | Lasby, Mike Lazarevich, Ivan Sinnadurai, Nish Lie, Sean Ioannou, Yani Thangarasa, Vithursan |
| author_facet | Lasby, Mike Lazarevich, Ivan Sinnadurai, Nish Lie, Sean Ioannou, Yani Thangarasa, Vithursan |
| contents | Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13999 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | REAP the Experts: Why Pruning Prevails for One-Shot MoE compression Lasby, Mike Lazarevich, Ivan Sinnadurai, Nish Lie, Sean Ioannou, Yani Thangarasa, Vithursan Machine Learning Artificial Intelligence I.2.7 Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts. |
| title | REAP the Experts: Why Pruning Prevails for One-Shot MoE compression |
| topic | Machine Learning Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2510.13999 |