Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2511.02237 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908626674778112 |
|---|---|
| author | Oncescu, Costin-Andrei Wu, Qingyang Chung, Wai Tong Wu, Robert Gopal, Bryan Wang, Junxiong Dao, Tri Athiwaratkun, Ben |
| author_facet | Oncescu, Costin-Andrei Wu, Qingyang Chung, Wai Tong Wu, Robert Gopal, Bryan Wang, Junxiong Dao, Tri Athiwaratkun, Ben |
| contents | An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for dynamically re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a batch-aware routing that works by having tokens piggyback experts that have already been loaded into memory due to being crucial to other tokens within the same batch. Empirically, we evaluate our method on the Qwen3-30B and Qwen3-235B models with a batch size of $16$. Without any statistically significant loss in accuracy, our approach achieves latency reductions of $39\%$ and $15\%$ in the MoE layer decode latency, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02237 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining Oncescu, Costin-Andrei Wu, Qingyang Chung, Wai Tong Wu, Robert Gopal, Bryan Wang, Junxiong Dao, Tri Athiwaratkun, Ben Machine Learning An increasing number of LLMs employ Mixture-of-Experts (MoE) architectures where the feed-forward layer is replaced by a pool of experts and each token only activates a small subset of them. During autoregressive generation, these models often enter a memory-bound regime even for moderate batch sizes because the average expert load grows more slowly than in an equivalent dense feedforward layer. Consequently, MoE latency is governed by the number of activated experts. We introduce a framework for dynamically re-routing token-to-expert mapping to lower this number (and thus, the decode latency) while preserving a comparable quality. Our best results use a batch-aware routing that works by having tokens piggyback experts that have already been loaded into memory due to being crucial to other tokens within the same batch. Empirically, we evaluate our method on the Qwen3-30B and Qwen3-235B models with a batch size of $16$. Without any statistically significant loss in accuracy, our approach achieves latency reductions of $39\%$ and $15\%$ in the MoE layer decode latency, respectively. |
| title | Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2511.02237 |