Copertina

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhang, Zeliang, Liu, Xiaodong, Cheng, Hao, Xu, Chenliang, Gao, Jianfeng
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Training Large Reasoning Models Efficiently via Progressive Thought Encoding
di: Zhang, Zeliang, et al.
Pubblicazione: (2026)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
di: Lu, Xudong, et al.
Pubblicazione: (2024)

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
di: Ahrac, Sagi, et al.
Pubblicazione: (2026)

SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
di: Muzio, Alexandre, et al.
Pubblicazione: (2024)

CoSMoEs: Compact Sparse Mixture of Experts
di: Huber, Patrick, et al.
Pubblicazione: (2025)

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
di: Song, Chenyang, et al.
Pubblicazione: (2026)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
di: Dong, Zican, et al.
Pubblicazione: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
di: Kim, Junhyuck, et al.
Pubblicazione: (2026)

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
di: He, Haoze, et al.
Pubblicazione: (2026)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
di: Wang, Zihan, et al.
Pubblicazione: (2025)

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
di: Gao, Shangqian, et al.
Pubblicazione: (2025)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
di: Huang, Wei, et al.
Pubblicazione: (2024)

$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
di: Chen, Guanjie, et al.
Pubblicazione: (2024)

$μ$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
di: Koike-Akino, Toshiaki, et al.
Pubblicazione: (2025)

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
di: Zhuang, Haomin, et al.
Pubblicazione: (2024)

Mixture of Lookup Experts
di: Jie, Shibo, et al.
Pubblicazione: (2025)

Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
di: Wang, Lean, et al.
Pubblicazione: (2024)

Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
di: Piękos, Piotr, et al.
Pubblicazione: (2025)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
di: Zheng, Chen, et al.
Pubblicazione: (2025)

CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts
di: Su, Zhenpeng, et al.
Pubblicazione: (2024)

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
di: Lv, Ang, et al.
Pubblicazione: (2025)

$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
di: Takashiro, Shota, et al.
Pubblicazione: (2026)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
di: Kim, Gyeongman, et al.
Pubblicazione: (2025)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
di: Pan, Bowen, et al.
Pubblicazione: (2024)

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
di: Zhang, Di, et al.
Pubblicazione: (2025)

MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
di: Xie, Yanyue, et al.
Pubblicazione: (2024)

Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
di: Sarkar, Soumajyoti, et al.
Pubblicazione: (2024)

HMoE: Heterogeneous Mixture of Experts for Language Modeling
di: Wang, An, et al.
Pubblicazione: (2024)

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
di: Song, Guanghui, et al.
Pubblicazione: (2025)

PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning
di: Lin, Zhisheng, et al.
Pubblicazione: (2024)

Routing-Free Mixture-of-Experts
di: Liu, Yilun, et al.
Pubblicazione: (2026)

Bayesian Mixture of Experts For Large Language Models
di: Dialameh, Maryam, et al.
Pubblicazione: (2025)

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
di: Nakamura, Taishi, et al.
Pubblicazione: (2025)

Mixture of Experts Made Intrinsically Interpretable
di: Yang, Xingyi, et al.
Pubblicazione: (2025)

Maximum Score Routing For Mixture-of-Experts
di: Dong, Bowen, et al.
Pubblicazione: (2025)

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
di: Nakamura, Taishi, et al.
Pubblicazione: (2025)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
di: Yoon, Youngsik, et al.
Pubblicazione: (2026)

XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
di: Yang, Yuanhang, et al.
Pubblicazione: (2024)

Mixture of LoRA Experts
di: Wu, Xun, et al.
Pubblicazione: (2024)

Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning
di: Siddika, Fatema, et al.
Pubblicazione: (2026)