Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.00254 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915971763011584 |
|---|---|
| author | Choi, Junsun Son, Sam Choi, Sunjin Kim, Hansung Shao, Yakun Sophia Shenker, Scott Ratnasamy, Sylvia Nikolic, Borivoje |
| author_facet | Choi, Junsun Son, Sam Choi, Sunjin Kim, Hansung Shao, Yakun Sophia Shenker, Scott Ratnasamy, Sylvia Nikolic, Borivoje |
| contents | Mixture-of-experts (MoE) architectures have turned LLM serving into a cluster-scale workload in which communication consumes a considerable portion of LLM serving runtime. This has prompted industry to invest heavily in expensive high-bandwidth scale-up networks. We question whether such costly infrastructure is strictly necessary. We present the first systematic cross-layer analysis of network cost-effectiveness for MoE LLM serving, comparing four representative XPU (e.g., GPU/TPU) topologies (scale-up, scale-out, 3D torus, and 3D full-mesh). We find that lower-cost switchless topologies are more cost-effective than the scale-up topology across all serving scenarios explored, improving cost-effectiveness by 20.6-56.2%. In particular, the 3D full-mesh topology is Pareto-optimal in terms of the performance-cost tradeoff. We also find that current scale-up link bandwidths are over-provisioned: reducing the link bandwidth improves throughput per cost by up to 27%. A forward-looking analysis of upcoming GPU generations indicates that the cost-performance advantage of switchless networks will likely persist. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_00254 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving Choi, Junsun Son, Sam Choi, Sunjin Kim, Hansung Shao, Yakun Sophia Shenker, Scott Ratnasamy, Sylvia Nikolic, Borivoje Networking and Internet Architecture Artificial Intelligence Mixture-of-experts (MoE) architectures have turned LLM serving into a cluster-scale workload in which communication consumes a considerable portion of LLM serving runtime. This has prompted industry to invest heavily in expensive high-bandwidth scale-up networks. We question whether such costly infrastructure is strictly necessary. We present the first systematic cross-layer analysis of network cost-effectiveness for MoE LLM serving, comparing four representative XPU (e.g., GPU/TPU) topologies (scale-up, scale-out, 3D torus, and 3D full-mesh). We find that lower-cost switchless topologies are more cost-effective than the scale-up topology across all serving scenarios explored, improving cost-effectiveness by 20.6-56.2%. In particular, the 3D full-mesh topology is Pareto-optimal in terms of the performance-cost tradeoff. We also find that current scale-up link bandwidths are over-provisioned: reducing the link bandwidth improves throughput per cost by up to 27%. A forward-looking analysis of upcoming GPU generations indicates that the cost-performance advantage of switchless networks will likely persist. |
| title | Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving |
| topic | Networking and Internet Architecture Artificial Intelligence |
| url | https://arxiv.org/abs/2605.00254 |