Copertina

Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Li, Junzhuo, Jiang, Peijie, Tian, Changxin, Liu, Jia, Zhang, Zhiqiang, Hu, Xuming
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
di: Li, Junzhuo, et al.
Pubblicazione: (2025)

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
di: Li, Junzhuo, et al.
Pubblicazione: (2025)

Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
di: Chu, Kexin, et al.
Pubblicazione: (2025)

AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
di: Gao, Yuting, et al.
Pubblicazione: (2025)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
di: Li, Cheng, et al.
Pubblicazione: (2025)

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
di: Liu, Baihui, et al.
Pubblicazione: (2026)

Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
di: Abnar, Samira, et al.
Pubblicazione: (2025)

Efficiently Editing Mixture-of-Experts Models with Compressed Experts
di: He, Yifei, et al.
Pubblicazione: (2025)

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
di: Zhuang, Haomin, et al.
Pubblicazione: (2024)

MC#: Mixture Compressor for Mixture-of-Experts Large Models
di: Huang, Wei, et al.
Pubblicazione: (2025)

Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
di: Liang, Jingcong, et al.
Pubblicazione: (2025)

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts
di: Xie, Zhitian, et al.
Pubblicazione: (2024)

How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
di: Park, Sumin, et al.
Pubblicazione: (2025)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
di: Zhao, Guoliang, et al.
Pubblicazione: (2025)

HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
di: Wei, Jia, et al.
Pubblicazione: (2026)

Scaling Laws for Fine-Grained Mixture of Experts
di: Krajewski, Jakub, et al.
Pubblicazione: (2024)

Generalization and Scaling Laws for Mixture-of-Experts Transformers
di: Mayaki, Mansour Zoubeirou a
Pubblicazione: (2026)

Neural Inhibition Improves Dynamic Routing and Mixture of Experts
di: Zou, Will Y., et al.
Pubblicazione: (2025)

Mixture of Experts in Large Language Models
di: Zhang, Danyang, et al.
Pubblicazione: (2025)

Speculating Experts Accelerates Inference for Mixture-of-Experts
di: Madan, Vivan, et al.
Pubblicazione: (2026)

Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
di: Yan, Jiaming, et al.
Pubblicazione: (2025)

Wavelet Mixture of Experts for Time Series Forecasting
di: Zhou, Zheng, et al.
Pubblicazione: (2025)

Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
di: Nguyen-Nhat, Minh-Khoi, et al.
Pubblicazione: (2025)

Mixture of Diverse Size Experts
di: Sun, Manxi, et al.
Pubblicazione: (2024)

Mixture of Weak & Strong Experts on Graphs
di: Zeng, Hanqing, et al.
Pubblicazione: (2023)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
di: Lu, Xudong, et al.
Pubblicazione: (2024)

Mixture of Raytraced Experts
di: Perin, Andrea, et al.
Pubblicazione: (2025)

Mixture-of-Experts Meets In-Context Reinforcement Learning
di: Wu, Wenhao, et al.
Pubblicazione: (2025)

MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
di: Yang, Cheng, et al.
Pubblicazione: (2024)

LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing
di: Hao, Jiawei, et al.
Pubblicazione: (2026)

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
di: Nakamura, Taishi, et al.
Pubblicazione: (2025)

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
di: Guo, Yongxin, et al.
Pubblicazione: (2024)

Dynamic Mixture of Experts Against Severe Distribution Shifts
di: Kim, Donghu
Pubblicazione: (2025)

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
di: Dwivedi, Chaitanya, et al.
Pubblicazione: (2026)

Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
di: Li, Albus Yizhuo, et al.
Pubblicazione: (2026)

Multilingual Routing in Mixture-of-Experts
di: Bandarkar, Lucas, et al.
Pubblicazione: (2025)

Sparsity and Superposition in Mixture of Experts
di: Chaudhari, Marmik, et al.
Pubblicazione: (2025)

Mixture of Concept Bottleneck Experts
di: De Santis, Francesco, et al.
Pubblicazione: (2026)

Mixture of A Million Experts
di: He, Xu Owen
Pubblicazione: (2024)

Mixture of Experts in a Mixture of RL settings
di: Willi, Timon, et al.
Pubblicazione: (2024)