Image de couverture de livre

Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhou, Yixiao, Zhao, Ziyu, Cheng, Dongzhou, wu, zhiliang, Gui, Jie, Yang, Yi, Wu, Fei, Cheng, Yu, Fan, Hehe
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement
par: Zhou, Yixiao, et autres
Publié: (2026)

Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL
par: Zhou, Yixiao, et autres
Publié: (2026)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
par: Zhang, Zeliang, et autres
Publié: (2024)

Mixture of Neuron Experts
par: Cheng, Runxi, et autres
Publié: (2025)

Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
par: Zhao, Ziyu, et autres
Publié: (2025)

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
par: Nakamura, Taishi, et autres
Publié: (2025)

Mixture-of-Experts with Gradient Conflict-Driven Subspace Topology Pruning for Emergent Modularity
par: Gan, Yuxing, et autres
Publié: (2025)

A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
par: Chowdhury, Mohammed Nowaz Rabbani, et autres
Publié: (2024)

Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
par: Han, Yixuan, et autres
Publié: (2025)

DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning
par: Bai, Sikai, et autres
Publié: (2025)

MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
par: Yang, Cheng, et autres
Publié: (2024)

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
par: Liu, Enshu, et autres
Publié: (2024)

Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts
par: Zhang, Yue, et autres
Publié: (2025)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
par: Lu, Xudong, et autres
Publié: (2024)

ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
par: Zhao, Ziyu, et autres
Publié: (2026)

Dynamic Adaptive Shared Experts with Grouped Multi-Head Attention Mixture of Experts
par: Li, Cheng, et autres
Publié: (2025)

Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs
par: Jiang, Yukun, et autres
Publié: (2026)

Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers
par: Pavlitska, Svetlana, et autres
Publié: (2025)

Layerwise Recurrent Router for Mixture-of-Experts
par: Qiu, Zihan, et autres
Publié: (2024)

Expert Merging in Sparse Mixture of Experts with Nash Bargaining
par: Nguyen, Dung V., et autres
Publié: (2025)

Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
par: Guo, Hongcheng, et autres
Publié: (2025)

Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
par: Jaiswal, Ajay, et autres
Publié: (2025)

Opportunistic Expert Activation: Batch-Aware Expert Routing for Faster Decode Without Retraining
par: Oncescu, Costin-Andrei, et autres
Publié: (2025)

$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
par: Chen, Guanjie, et autres
Publié: (2024)

Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
par: Hu, Wentao, et autres
Publié: (2025)

Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning
par: Zhao, Ziyu, et autres
Publié: (2024)

A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
par: Liu, Zijie, et autres
Publié: (2026)

TradExpert: Revolutionizing Trading with Mixture of Expert LLMs
par: Ding, Qianggang, et autres
Publié: (2024)

From Sparse to Soft Mixtures of Experts
par: Puigcerver, Joan, et autres
Publié: (2023)

FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting
par: Liu, Fan, et autres
Publié: (2025)

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
par: He, Haoze, et autres
Publié: (2026)

Routing-Free Mixture-of-Experts
par: Liu, Yilun, et autres
Publié: (2026)

Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
par: Nguyen-Nhat, Minh-Khoi, et autres
Publié: (2025)

Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
par: Nikolic, Strahinja, et autres
Publié: (2025)

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
par: Ahrac, Sagi, et autres
Publié: (2026)

Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts
par: Yuan, Yike, et autres
Publié: (2025)

Mixture of Length and Pruning Experts for Knowledge Graphs Reasoning
par: Du, Enjun, et autres
Publié: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
par: Kim, Junhyuck, et autres
Publié: (2026)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
par: Dong, Zican, et autres
Publié: (2025)

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
par: Zhuang, Haomin, et autres
Publié: (2024)