Imagen de Portada

Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Li, Houyi, Lo, Ka Man, Xuyang, Shijie, Wang, Ziqi, Zheng, Wenzhen, Zhang, Haocheng, Li, Zhao, Zhou, Shuigeng, Zhang, Xiangyu, Jiang, Daxin
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
por: Li, Houyi, et al.
Publicado: (2025)

Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
por: Li, Houyi, et al.
Publicado: (2025)

Multi-matrix Factorization Attention
por: Hu, Jingcheng, et al.
Publicado: (2024)

Is Compression Really Linear with Code Intelligence?
por: Xuyang, Shijie, et al.
Publicado: (2025)

A Closer Look into Mixture-of-Experts in Large Language Models
por: Lo, Ka Man, et al.
Publicado: (2024)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
por: Lu, Xudong, et al.
Publicado: (2024)

Mixture of Experts for Low-Resource LLMs
por: Joseph, Ori Bar, et al.
Publicado: (2026)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
por: Huang, Wei, et al.
Publicado: (2024)

Multi-Task Dense Prediction via Mixture of Low-Rank Experts
por: Yang, Yuqi, et al.
Publicado: (2024)

Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking
por: Zhu, Yabin, et al.
Publicado: (2026)

An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint
por: Sun, Yi, et al.
Publicado: (2025)

Investigating Mixture of Experts in Dense Retrieval
por: Sokli, Effrosyni, et al.
Publicado: (2024)

Frequency-Adaptive Pan-Sharpening with Mixture of Experts
por: He, Xuanhua, et al.
Publicado: (2024)

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
por: Zhuang, Haomin, et al.
Publicado: (2024)

Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs
por: Jiang, Yukun, et al.
Publicado: (2026)

Mixture of Experts Approaches in Dense Retrieval Tasks
por: Sokli, Effrosyni, et al.
Publicado: (2025)

Scaling Laws for Code: A More Data-Hungry Regime
por: Luo, Xianzhen, et al.
Publicado: (2025)

Fast Causal Discovery by Approximate Kernel-based Generalized Score Functions with Linear Computational Complexity
por: Ren, Yixin, et al.
Publicado: (2024)

Asymptotic behavior of the Speed of Sound in Dense Matter
por: Shukla, Udita, et al.
Publicado: (2025)

MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
por: Chen, Xiaodong, et al.
Publicado: (2025)

Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs
por: Li, Zhongyang, et al.
Publicado: (2025)

Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs
por: Bai, Jun, et al.
Publicado: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
por: Kim, Junhyuck, et al.
Publicado: (2026)

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
por: Panda, Ashwinee, et al.
Publicado: (2025)

FreqMoE: Enhancing Time Series Forecasting through Frequency Decomposition Mixture of Experts
por: Liu, Ziqi
Publicado: (2025)

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
por: Gao, Shangqian, et al.
Publicado: (2025)

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
por: Hui, Tingfeng, et al.
Publicado: (2024)

From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
por: Ma, Xiangyu, et al.
Publicado: (2026)

HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing
por: Huang, Haochen, et al.
Publicado: (2025)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
por: Pan, Bowen, et al.
Publicado: (2024)

Surpassing Quantum Noise Limits with Nonlinear Amplification
por: Ren, Ya-Long, et al.
Publicado: (2026)

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
por: Li, Yunxin, et al.
Publicado: (2024)

An Entailment Tree Generation Approach for Multimodal Multi-Hop Question Answering with Mixture-of-Experts and Iterative Feedback Mechanism
por: Zhang, Qing, et al.
Publicado: (2024)

Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
por: Wong, Man Yung
Publicado: (2026)

MalMoE: Mixture-of-Experts Enhanced Encrypted Malicious Traffic Detection Under Graph Drift
por: Tan, Yunpeng, et al.
Publicado: (2026)

Do LLMs Surpass Encoders for Biomedical NER?
por: Obeidat, Motasem S, et al.
Publicado: (2025)

TradExpert: Revolutionizing Trading with Mixture of Expert LLMs
por: Ding, Qianggang, et al.
Publicado: (2024)

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
por: Wu, Haoyuan, et al.
Publicado: (2024)

Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
por: Li, Lujun, et al.
Publicado: (2025)

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
por: Sukhbaatar, Sainbayar, et al.
Publicado: (2024)