Copertina

A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Liu, Zijie, Peng, Jie, Duan, Jinhao, Liu, Zirui, Zhou, Kaixiong, Liang, Mingfu, Simon, Luke, Liu, Xi, Xu, Zhaozhuo, Chen, Tianlong
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing
di: Shahout, Rana, et al.
Pubblicazione: (2025)

GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
di: Deng, Jianing, et al.
Pubblicazione: (2026)

MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
di: Zhang, Jinhao, et al.
Pubblicazione: (2025)

Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
di: Wang, Lean, et al.
Pubblicazione: (2024)

Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts
di: Nguyen, Xuan-Phi, et al.
Pubblicazione: (2026)

Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs
di: Liu, Jiazhen, et al.
Pubblicazione: (2025)

Load Balancing Mixture of Experts with Similarity Preserving Routers
di: Omi, Nabil, et al.
Pubblicazione: (2025)

QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
di: Li, Pingzhi, et al.
Pubblicazione: (2024)

Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
di: Chu, Kexin, et al.
Pubblicazione: (2025)

Hexa-MoE: Efficient and Heterogeneous-aware Training for Mixture-of-Experts
di: Luo, Shuqing, et al.
Pubblicazione: (2024)

Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models
di: Wu, Yuheng, et al.
Pubblicazione: (2025)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
di: Qiu, Zihan, et al.
Pubblicazione: (2025)

$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
di: Chen, Guanjie, et al.
Pubblicazione: (2024)

Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
di: Zhou, Yixiao, et al.
Pubblicazione: (2025)

Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
di: Chowdhury, Mohammed Nowaz Rabbani, et al.
Pubblicazione: (2026)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
di: Huang, Wei, et al.
Pubblicazione: (2024)

FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting
di: Liu, Fan, et al.
Pubblicazione: (2025)

TradExpert: Revolutionizing Trading with Mixture of Expert LLMs
di: Ding, Qianggang, et al.
Pubblicazione: (2024)

Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models
di: Sun, Yuan
Pubblicazione: (2025)

$ϕ$-Balancing for Mixture-of-Experts Training
di: Chen, Lizhang, et al.
Pubblicazione: (2026)

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
di: Zhuang, Haomin, et al.
Pubblicazione: (2024)

Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs
di: Jiang, Yukun, et al.
Pubblicazione: (2026)

A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
di: Han, X. Y., et al.
Pubblicazione: (2025)

Can GRPO Help LLMs Transcend Their Pretraining Origin?
di: Ni, Kangqi, et al.
Pubblicazione: (2025)

RoME: Domain-Robust Mixture-of-Experts for MILP Solution Prediction across Domains
di: Pu, Tianle, et al.
Pubblicazione: (2025)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
di: Zhang, Zeliang, et al.
Pubblicazione: (2024)

Sparse Neurons Carry Strong Signals of Question Ambiguity in LLMs
di: Zhang, Zhuoxuan, et al.
Pubblicazione: (2025)

Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Training
di: Mouzouni, Charafeddine
Pubblicazione: (2026)

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
di: Liu, Zirui, et al.
Pubblicazione: (2024)

MoPEQ: Mixture of Mixed Precision Quantized Experts
di: Chitty-Venkata, Krishna Teja, et al.
Pubblicazione: (2025)

Dialogue is Better Than Monologue: Instructing Medical LLMs via Strategical Conversations
di: Liu, Zijie, et al.
Pubblicazione: (2025)

Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation
di: Xue, Bowen, et al.
Pubblicazione: (2025)

Enhancing Online Continual Learning with Plug-and-Play State Space Model and Class-Conditional Mixture of Discretization
di: Liu, Sihao, et al.
Pubblicazione: (2024)

MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts
di: Wang, Wenfeng, et al.
Pubblicazione: (2025)

RailS: Load Balancing for All-to-All Communication in Distributed Mixture-of-Experts Training
di: Xu, Heng, et al.
Pubblicazione: (2025)

Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
di: Yang, Jiajie
Pubblicazione: (2025)

NFCDS: A Plug-and-Play Noise Frequency-Controlled Diffusion Sampling Strategy for Image Restoration
di: Wang, Zhen, et al.
Pubblicazione: (2026)

Plug-and-Play Context Feature Reuse for Efficient Masked Generation
di: Liu, Xuejie, et al.
Pubblicazione: (2025)

I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
di: Xin, Jiayi, et al.
Pubblicazione: (2025)

Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs
di: Bai, Jun, et al.
Pubblicazione: (2025)