Copertina

DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Huang, Yongqi, Ye, Peng, Huang, Chenyu, Cao, Jianjian, Zhang, Lin, Li, Baopu, Yu, Gang, Chen, Tao
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
di: Dwivedi, Chaitanya, et al.
Pubblicazione: (2026)

BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts
di: Zhang, Qizhen, et al.
Pubblicazione: (2024)

Upcycling Large Language Models into Mixture of Experts
di: He, Ethan, et al.
Pubblicazione: (2024)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
di: Liew, Seng Pei, et al.
Pubblicazione: (2025)

Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts
di: Chen, Shengzhuang, et al.
Pubblicazione: (2025)

CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcycling
di: Wang, Xinze, et al.
Pubblicazione: (2025)

MoIN: Mixture of Introvert Experts to Upcycle an LLM
di: Tejankar, Ajinkya, et al.
Pubblicazione: (2024)

Mixture of Length and Pruning Experts for Knowledge Graphs Reasoning
di: Du, Enjun, et al.
Pubblicazione: (2025)

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
di: Hui, Tingfeng, et al.
Pubblicazione: (2024)

Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling
di: Ran, Junfeng, et al.
Pubblicazione: (2025)

Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
di: Fang, Zhiyuan, et al.
Pubblicazione: (2025)

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
di: Guo, Yongxin, et al.
Pubblicazione: (2024)

Efficiently Editing Mixture-of-Experts Models with Compressed Experts
di: He, Yifei, et al.
Pubblicazione: (2025)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
di: Lu, Xudong, et al.
Pubblicazione: (2024)

MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
di: Teo, Rachel S. Y., et al.
Pubblicazione: (2025)

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
di: Nakamura, Taishi, et al.
Pubblicazione: (2025)

MC#: Mixture Compressor for Mixture-of-Experts Large Models
di: Huang, Wei, et al.
Pubblicazione: (2025)

MoFE-Time: Mixture of Frequency Domain Experts for Time-Series Forecasting Models
di: Liu, Yiwen, et al.
Pubblicazione: (2025)

PreMoE: Proactive Inference for Efficient Mixture-of-Experts
di: Pei, Zehua, et al.
Pubblicazione: (2025)

Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
di: Shen, Li, et al.
Pubblicazione: (2024)

XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
di: Ding, Yifeng, et al.
Pubblicazione: (2024)

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
di: Zhang, Di, et al.
Pubblicazione: (2025)

Towards Generalization-Oriented Models for Vehicle Routing Problems with Mixture-of-Experts
di: Miao, Changhao, et al.
Pubblicazione: (2026)

Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping
di: Huang, Ziye, et al.
Pubblicazione: (2024)

MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
di: Ai, Mengting, et al.
Pubblicazione: (2023)

MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
di: Cai, Weilin, et al.
Pubblicazione: (2024)

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
di: Liu, Enshu, et al.
Pubblicazione: (2024)

Toward Inference-optimal Mixture-of-Expert Large Language Models
di: Yun, Longfei, et al.
Pubblicazione: (2024)

AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
di: Gao, Yuting, et al.
Pubblicazione: (2025)

MoNTA: Accelerating Mixture-of-Experts Training with Network-Traffc-Aware Parallel Optimization
di: Guo, Jingming, et al.
Pubblicazione: (2024)

$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
di: Chen, Guanjie, et al.
Pubblicazione: (2024)

Mixture of LoRA Experts
di: Wu, Xun, et al.
Pubblicazione: (2024)

EMR-Merging: Tuning-Free High-Performance Model Merging
di: Huang, Chenyu, et al.
Pubblicazione: (2024)

MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
di: Liu, Zehua, et al.
Pubblicazione: (2025)

Less is More: Undertraining Experts Improves Model Upcycling
di: Horoi, Stefan, et al.
Pubblicazione: (2025)

Training-Free Dynamic Upcycling of Expert Language Models
di: Fanì, Eros, et al.
Pubblicazione: (2026)

Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach
di: Chen, Xiaobing, et al.
Pubblicazione: (2025)

Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression
di: Tang, Yuntian, et al.
Pubblicazione: (2026)

DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
di: Aghdam, Maryam Akhavan, et al.
Pubblicazione: (2024)

UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
di: Huang, Minbin, et al.
Pubblicazione: (2026)