Imagen de Portada

On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Nguyen, Huy, Doan, Thong T., Pham, Quang, Bui, Nghi D. Q., Ho, Nhat, Rinaldo, Alessandro
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2024)

Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function
por: Pham, Tuan Minh, et al.
Publicado: (2026)

Convergence Rates for Softmax Gating Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2025)

On Least Square Estimation in Softmax Gating Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2024)

On Bayesian Softmax-Gated Mixture-of-Experts Models
por: Bariletto, Nicola, et al.
Publicado: (2026)

A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
por: Nguyen, Viet, et al.
Publicado: (2026)

Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
por: Yan, Fanqi, et al.
Publicado: (2025)

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
por: Dai, Damai, et al.
Publicado: (2024)

CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition
por: Nguyen, Nam V., et al.
Publicado: (2025)

Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention
por: Akbarian, Pedram, et al.
Publicado: (2024)

Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2023)

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
por: Nguyen, Nam V., et al.
Publicado: (2024)

Statistical Advantages of Perturbing Cosine Router in Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2024)

On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
por: Yan, Fanqi, et al.
Publicado: (2025)

CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
por: Manh, Dung Nguyen, et al.
Publicado: (2024)

A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2023)

On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
por: Nguyen, Huy, et al.
Publicado: (2024)

FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
por: Han, Xing, et al.
Publicado: (2024)

Some species of macro-gastropods in the coastal zone of Khanh Hoa province
por: Bui, Quang Nghi
Publicado: (2020)

HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale
por: Phan, Huy Nhat, et al.
Publicado: (2024)

On Parameter Estimation in Deviated Gaussian Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2024)

Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts
por: Le, Minh, et al.
Publicado: (2024)

Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity
por: Yan, Fanqi, et al.
Publicado: (2026)

CodeFlow: Program Behavior Prediction with Dynamic Dependencies Learning
por: Le, Cuong Chi, et al.
Publicado: (2024)

Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
por: Nguyen, Huy, et al.
Publicado: (2024)

SAGE: Shape-Adapting Gated Experts for Adaptive Histopathology Image Segmentation
por: Thai, Gia Huy, et al.
Publicado: (2025)

New record of the Ophuroid Ophiosphaera insignis Brock, 1888 (Ophiuroidea- Echinodermata) in Vietnamese seawaters.
por: Nguyen, Thi My Ngan, et al.
Publicado: (2015)

Description of Stichopus sp. (phylum Echinodermata – class Holothuroidea) collected in Nha Trang bay
por: Nguyen, Thi My Ngan, et al.
Publicado: (2015)

Mixture of Experts Meets Prompt-Based Continual Learning
por: Le, Minh, et al.
Publicado: (2024)

Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
por: Nguyen, Huy, et al.
Publicado: (2023)

Climatological regime and weather condition occurred on the cruise expedition (May 1999) on Vietnam continental shelf
por: Thong, Bui Xuan, et al.
Publicado: (2001)

Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned
por: Bui, Nghi D. Q.
Publicado: (2026)

CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
por: Pham, Quang, et al.
Publicado: (2024)

Conformalized Bayesian Inference, with Applications to Random Partition Models
por: Bariletto, Nicola, et al.
Publicado: (2025)

Direct Oxidation of H 2 S by Air at Low Temperatures Using Catalysts Derived from Industrial Solid Waste
por: Thanh Son Cam, et al.
Publicado: (2025)

Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts
por: Yan, Fanqi, et al.
Publicado: (2024)

Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts
por: Le, Minh, et al.
Publicado: (2025)

Tri‐reforming of Methane over a Hydroxyapatite‐Supported Nickel Catalyst Prepared by Cation Exchange
por: Xuan‐Huynh Pham, et al.
Publicado: (2025)

CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generation
por: Doan, Duy Tung, et al.
Publicado: (2026)

Frequent occurrence of tetrodotoxin in the marine gastropod Nassarius glans causing a food poisoning in Khanh Hoa province, Vietnam in 2020
por: Dao, Viet Ha, et al.
Publicado: (2023)