Image de couverture de livre

Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Farhat, Yehya, Shili, Hamza ElMokhtar, Liao, Fangshuo, Dun, Chen, Garcia, Mirian Hipolito, Zheng, Guoqing, Awadallah, Ahmed Hassan, Sim, Robert, Dimitriadis, Dimitrios, Kyrillidis, Anastasios
Format:	Preprint
Publié:	2023
Sujets:	Machine Learning Artificial Intelligence Optimization and Control
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation
par: Dun, Chen, et autres
Publié: (2023)

Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
par: Liao, Fangshuo, et autres
Publié: (2025)

TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
par: Menezes, Michael, et autres
Publié: (2025)

Provable Accelerated Convergence of Nesterov's Momentum for Deep ReLU Neural Networks
par: Liao, Fangshuo, et autres
Publié: (2023)

Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation
par: Liao, Fangshuo, et autres
Publié: (2025)

AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
par: Su, Barbara, et autres
Publié: (2026)

SGD at the Edge of Stability: The Stochastic Sharpness Gap
par: Liao, Fangshuo, et autres
Publié: (2026)

One Rank at a Time: Cascading Error Dynamics in Sequential Learning
par: Vandchali, Mahtab Alizadeh, et autres
Publié: (2025)

Projection-Free CNN Pruning via Frank-Wolfe with Momentum: Sparser Models with Less Pretraining
par: Shili, Hamza ElMokhtar, et autres
Publié: (2025)

On the Robustness of Decision-Focused Learning
par: Farhat, Yehya
Publié: (2023)

On the Error-Propagation of Inexact Hotelling's Deflation for Principal Component Analysis
par: Liao, Fangshuo, et autres
Publié: (2023)

Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
par: Mirvakhabova, Leyla, et autres
Publié: (2025)

Convergence Analysis of Two-Layer Neural Networks under Gaussian Input Masking
par: Kolomvaki, Afroditi, et autres
Publié: (2026)

Exploiting Low-Rank Structure in Max-K-Cut Problems
par: Stevens, Ria, et autres
Publié: (2026)

Advancing Expert Specialization for Better MoE
par: Guo, Hongcan, et autres
Publié: (2025)

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
par: Wang, Xi, et autres
Publié: (2026)

Exploring How LLMs Capture and Represent Domain-Specific Knowledge
par: Garcia, Mirian Hipolito, et autres
Publié: (2025)

One Model, Two Roles: Emergent Specialization in a Shared Recurrent Transformer
par: Shen, Jucheng, et autres
Publié: (2026)

DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
par: Wang, Jing, et autres
Publié: (2026)

Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
par: Ye, Charles, et autres
Publié: (2026)

SD-MoE: Spectral Decomposition for Effective Expert Specialization
par: Huang, Ruijun, et autres
Publié: (2026)

GHOST: Unmasking Phantom States in Mamba2 via Grouped Hidden-state Output-aware Selection & Truncation
par: Menezes, Michael, et autres
Publié: (2026)

Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
par: Jaiswal, Ajay, et autres
Publié: (2025)

GM-MoE: Low-Light Enhancement with Gated-Mechanism Mixture-of-Experts
par: Liao, Minwen, et autres
Publié: (2025)

TAG-MoE: Task-Aware Gating for Unified Generative Mixture-of-Experts
par: Xu, Yu, et autres
Publié: (2026)

Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
par: Hu, Rizhen, et autres
Publié: (2026)

SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
par: Bo, Zi-Hao, et autres
Publié: (2026)

AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference
par: Zhong, Shuzhang, et autres
Publié: (2024)

Using non-convex optimization in quantum process tomography: Factored gradient descent is tough to beat
par: Quiroga, David A., et autres
Publié: (2023)

Better Schedules for Low Precision Training of Deep Neural Networks
par: Wolfe, Cameron R., et autres
Publié: (2024)

Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts
par: Kang, Junmo, et autres
Publié: (2024)

MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
par: Falke, Tobias, et autres
Publié: (2026)

Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
par: Wu, Haoyuan, et autres
Publié: (2025)

EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
par: Qian, Yulei, et autres
Publié: (2024)

A Unified Kantorovich Duality for Multimarginal Optimal Transport
par: Cheryala, Yehya, et autres
Publié: (2026)

MoE-MLoRA for Multi-Domain CTR Prediction: Efficient Adaptation with Expert Specialization
par: Yaggel, Ken, et autres
Publié: (2025)

Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptation
par: Li, Junzhuo, et autres
Publié: (2025)

Quantum EigenGame for excited state calculation
par: Quiroga, David, et autres
Publié: (2025)

Sparse Crosscoders for diffing MoEs and Dense models
par: Chaudhari, Marmik, et autres
Publié: (2026)

OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
par: Wang, Liujianfu, et autres
Publié: (2025)