Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918467699998720 |
|---|---|
| author | Bambhaniya, Abhimanyu Jeong, Geonhwa Park, Jason Yu, Jiecao Lee, Jaewon Wang, Pengchao Kim, Changkyu Tang, Chunqiang Krishna, Tushar |
| author_facet | Bambhaniya, Abhimanyu Jeong, Geonhwa Park, Jason Yu, Jiecao Lee, Jaewon Wang, Pengchao Kim, Changkyu Tang, Chunqiang Krishna, Tushar |
| contents | Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling higher-quality outputs at manageable serving costs. However, MoE inference at scale is fundamentally bottlenecked by expert load imbalance and inefficient token routing, especially in multi-node deployments where tokens are not guaranteed to be routed to local experts, resulting in significant inter-node all-to-all communication overhead.
To systematically characterize these challenges, we profile SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B, on various datasets and collected over 100k real expert activation traces. Upon studying the expert activation patterns, we uncover various persistent properties across all the frontier MoE models: variable expert load imbalance, domain-specific expert activation where expert popularity shifts across task families (code, math, chat, general), and a strong correlation between prefill and decode expert activations. Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy to maximize token locality to the destination expert, thereby reducing inter-node communication. Across models and datasets, these optimizations help reduce all2all communication data up to 20, resulting in lower MoE decode latency and better accelerator utilization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_23150 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns Bambhaniya, Abhimanyu Jeong, Geonhwa Park, Jason Yu, Jiecao Lee, Jaewon Wang, Pengchao Kim, Changkyu Tang, Chunqiang Krishna, Tushar Machine Learning Artificial Intelligence Hardware Architecture Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling higher-quality outputs at manageable serving costs. However, MoE inference at scale is fundamentally bottlenecked by expert load imbalance and inefficient token routing, especially in multi-node deployments where tokens are not guaranteed to be routed to local experts, resulting in significant inter-node all-to-all communication overhead. To systematically characterize these challenges, we profile SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B, on various datasets and collected over 100k real expert activation traces. Upon studying the expert activation patterns, we uncover various persistent properties across all the frontier MoE models: variable expert load imbalance, domain-specific expert activation where expert popularity shifts across task families (code, math, chat, general), and a strong correlation between prefill and decode expert activations. Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy to maximize token locality to the destination expert, thereby reducing inter-node communication. Across models and datasets, these optimizations help reduce all2all communication data up to 20, resulting in lower MoE decode latency and better accelerator utilization. |
| title | Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns |
| topic | Machine Learning Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2604.23150 |