Imagen de Portada

Towards an empirical understanding of MoE design choices

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Fan, Dongyang, Messmer, Bettina, Jaggi, Martin
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Artificial Intelligence Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists
por: Fan, Dongyang, et al.
Publicado: (2024)

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection
por: Turki, Yassine, et al.
Publicado: (2026)

Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
por: Fan, Dongyang, et al.
Publicado: (2025)

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
por: Messmer, Bettina, et al.
Publicado: (2025)

GRIN: GRadient-INformed MoE
por: Liu, Liyuan, et al.
Publicado: (2024)

MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
por: Li, Shuhuai, et al.
Publicado: (2026)

Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
por: Wang, Bo, et al.
Publicado: (2026)

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
por: Li, Junzhuo, et al.
Publicado: (2025)

DoGE: Domain Reweighting with Generalization Estimation
por: Fan, Simin, et al.
Publicado: (2023)

Post-Trained MoE Can Skip Half Experts via Self-Distillation
por: Lv, Xingtai, et al.
Publicado: (2026)

LocMoE: A Low-Overhead MoE for Large Language Model Training
por: Li, Jing, et al.
Publicado: (2024)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
por: Zeng, Zhiyuan, et al.
Publicado: (2024)

Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
por: Ma, Wenhan, et al.
Publicado: (2025)

MESA: Improving MoE Safety Alignment via Decentralized Expertise
por: Sun, Yitong, et al.
Publicado: (2026)

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
por: Gu, Naibin, et al.
Publicado: (2025)

Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptation
por: Yang, Yuxin, et al.
Publicado: (2026)

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
por: Pióro, Maciej, et al.
Publicado: (2024)

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
por: Ludziejewski, Jan, et al.
Publicado: (2025)

$μ$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts
por: Koike-Akino, Toshiaki, et al.
Publicado: (2025)

Jakiro: Boosting Speculative Decoding with Decoupled Multi-Head via MoE
por: Huang, Haiduo, et al.
Publicado: (2025)

Leave It to the Experts: Detecting Knowledge Distillation via MoE Expert Signatures
por: Li, Pingzhi, et al.
Publicado: (2025)

Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
por: Bohne, Jason, et al.
Publicado: (2025)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
por: Tang, Shengkun, et al.
Publicado: (2026)

Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
por: Ye, Charles, et al.
Publicado: (2026)

Prediction Is All MoE Needs: Expert Load Distribution Goes from Fluctuating to Stabilizing
por: Cong, Peizhuang, et al.
Publicado: (2024)

Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM
por: ai, YuanLab., et al.
Publicado: (2026)

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
por: Liu, Baihui, et al.
Publicado: (2026)

MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE
por: Zibakhsh, Soheil, et al.
Publicado: (2025)

Continual Pre-training of MoEs: How robust is your router?
por: Thérien, Benjamin, et al.
Publicado: (2025)

Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
por: Krajewski, Jakub, et al.
Publicado: (2025)

Personalized Collaborative Fine-Tuning for On-Device Large Language Models
por: Wagner, Nicolas, et al.
Publicado: (2024)

MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
por: Xie, Yanyue, et al.
Publicado: (2024)

Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
por: Sun, Weigao, et al.
Publicado: (2025)

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
por: Liu, Yang, et al.
Publicado: (2026)

PithTrain: A Compact and Agent-Native MoE Training System
por: Lai, Ruihang, et al.
Publicado: (2026)

GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration
por: Bai, Ting, et al.
Publicado: (2024)

Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs
por: Bossy, Thierry, et al.
Publicado: (2025)

OrchMoE: Efficient Multi-Adapter Learning with Task-Skill Synergy
por: Wang, Haowen, et al.
Publicado: (2024)

Sequential choice in ordered bundles
por: Kohli, Rajeev, et al.
Publicado: (2024)

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
por: Zhu, Peijun, et al.
Publicado: (2025)