Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911615590334464 |
|---|---|
| author | Pei, Zehua Zhen, Hui-Ling Zou, Lancheng Yu, Xianzhi Liu, Wulong Pan, Sinno Jialin Yuan, Mingxuan Yu, Bei |
| author_facet | Pei, Zehua Zhen, Hui-Ling Zou, Lancheng Yu, Xianzhi Liu, Wulong Pan, Sinno Jialin Yuan, Mingxuan Yu, Bei |
| contents | Scaling large language models (LLMs) improves performance but significantly increases inference costs, with feed-forward networks (FFNs) consuming the majority of computational resources. While Mixture-of-Experts (MoE) architectures can reduce this cost through sparse activation, restructuring existing dense models into MoEs typically requires extensive retraining on hundreds of billions of tokens. We propose an analytical post-training framework that rapidly restructures FFNs into sparse MoE architectures using only a small calibration dataset. The method analyzes neuron activation patterns to partition neurons into always-active shared experts and conditionally activated routed experts, then constructs a router analytically from representative neuron statistics, enabling immediate deployment or optional lightweight fine-tuning. This approach applies both to dense models and recursively to existing MoE models for hierarchical sparsity. Experiments demonstrate up to $1.17\times$ speedup in compute-bound scenarios with only minutes of processing and 2k-sample fine-tuning, outperforming methods requiring orders of magnitude more resources. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04416 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis Pei, Zehua Zhen, Hui-Ling Zou, Lancheng Yu, Xianzhi Liu, Wulong Pan, Sinno Jialin Yuan, Mingxuan Yu, Bei Machine Learning Artificial Intelligence Scaling large language models (LLMs) improves performance but significantly increases inference costs, with feed-forward networks (FFNs) consuming the majority of computational resources. While Mixture-of-Experts (MoE) architectures can reduce this cost through sparse activation, restructuring existing dense models into MoEs typically requires extensive retraining on hundreds of billions of tokens. We propose an analytical post-training framework that rapidly restructures FFNs into sparse MoE architectures using only a small calibration dataset. The method analyzes neuron activation patterns to partition neurons into always-active shared experts and conditionally activated routed experts, then constructs a router analytically from representative neuron statistics, enabling immediate deployment or optional lightweight fine-tuning. This approach applies both to dense models and recursively to existing MoE models for hierarchical sparsity. Experiments demonstrate up to $1.17\times$ speedup in compute-bound scenarios with only minutes of processing and 2k-sample fine-tuning, outperforming methods requiring orders of magnitude more resources. |
| title | Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.04416 |