Imagen de Portada

Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wan, Weilin, Han, Jingtao, Zhang, Weizhong, Jin, Cheng
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Computational Budget Should Be Considered in Data Selection
por: Wan, Weilin, et al.
Publicado: (2025)

Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Training
por: Wan, Weilin, et al.
Publicado: (2025)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
por: Zhao, Guoliang, et al.
Publicado: (2025)

Scaling Laws for Optimal Data Mixtures
por: Shukor, Mustafa, et al.
Publicado: (2025)

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
por: Abnar, Samira, et al.
Publicado: (2025)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
por: Liew, Seng Pei, et al.
Publicado: (2025)

Generalization and Scaling Laws for Mixture-of-Experts Transformers
por: Mayaki, Mansour Zoubeirou a
Publicado: (2026)

Scaling Laws for Fine-Grained Mixture of Experts
por: Krajewski, Jakub, et al.
Publicado: (2024)

Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
por: Li, Junzhuo, et al.
Publicado: (2026)

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
por: Song, Chenyang, et al.
Publicado: (2026)

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
por: Zhu, Ruidong, et al.
Publicado: (2025)

Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
por: He, Shwai, et al.
Publicado: (2024)

On Linear Mode Connectivity of Mixture-of-Experts Architectures
por: Tran, Viet-Hoang, et al.
Publicado: (2025)

A Survey on Mixture of Experts in Large Language Models
por: Cai, Weilin, et al.
Publicado: (2024)

Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts
por: He, Shwai, et al.
Publicado: (2025)

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
por: Cai, Weilin, et al.
Publicado: (2024)

Scaling Machine Learning Interatomic Potentials with Mixtures of Experts
por: Liu, Yuzhi, et al.
Publicado: (2026)

MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
por: Cai, Weilin, et al.
Publicado: (2024)

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
por: Ludziejewski, Jan, et al.
Publicado: (2025)

MoFE: Mixture of Frozen Experts Architecture
por: Seo, Jean, et al.
Publicado: (2025)

What Scales in Cross-Entropy Scaling Law?
por: Yan, Junxi, et al.
Publicado: (2025)

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws
por: Jha, Nandan Kumar, et al.
Publicado: (2026)

Slicing and Dicing: Configuring Optimal Mixtures of Experts
por: Li, Margaret, et al.
Publicado: (2026)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
por: Zheng, Chen, et al.
Publicado: (2025)

BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
por: Song, Chenyang, et al.
Publicado: (2025)

Mixture of Online and Offline Experts for Non-stationary Time Series
por: Zhao, Zhilin, et al.
Publicado: (2022)

Adaptive Semantic Communication for Wireless Image Transmission Leveraging Mixture-of-Experts Mechanism
por: Wan, Haowen, et al.
Publicado: (2026)

How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
por: Park, Sumin, et al.
Publicado: (2025)

DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
por: Aghdam, Maryam Akhavan, et al.
Publicado: (2024)

UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
por: Huang, Minbin, et al.
Publicado: (2026)

Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
por: Li, Jingwei, et al.
Publicado: (2026)

Mixture of Lookup Experts
por: Jie, Shibo, et al.
Publicado: (2025)

FaST: Efficient and Effective Long-Horizon Forecasting for Large-Scale Spatial-Temporal Graphs via Mixture-of-Experts
por: Zhao, Yiji, et al.
Publicado: (2026)

On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
por: Nguyen, Huy, et al.
Publicado: (2024)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
por: Zhang, Zeliang, et al.
Publicado: (2024)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
por: Belenki, Lior, et al.
Publicado: (2025)

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
por: Lv, Ang, et al.
Publicado: (2025)

Dynamic Mixture-of-Experts for Incremental Graph Learning
por: Kong, Lecheng, et al.
Publicado: (2025)

How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
por: Vankadara, Leena Chennuru, et al.
Publicado: (2026)

Mixtures of Experts Unlock Parameter Scaling for Deep RL
por: Obando-Ceron, Johan, et al.
Publicado: (2024)