Imagen de Portada

FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Pan, Xinglin, Lin, Wenxiang, Zhang, Lin, Shi, Shaohuai, Tang, Zhenheng, Wang, Rui, Li, Bo, Chu, Xiaowen
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
por: Lin, Wenxiang, et al.
Publicado: (2025)

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
por: Pan, Xinglin, et al.
Publicado: (2024)

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
por: Pan, Xinglin, et al.
Publicado: (2025)

Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems
por: Lin, Wenxiang, et al.
Publicado: (2024)

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
por: Lin, Wenxiang, et al.
Publicado: (2026)

FedImpro: Measuring and Improving Client Update in Federated Learning
por: Tang, Zhenheng, et al.
Publicado: (2024)

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
por: Tang, Zhenheng, et al.
Publicado: (2024)

DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
por: Tang, Zhenheng, et al.
Publicado: (2025)

Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models
por: Dong, Peijie, et al.
Publicado: (2024)

ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
por: He, Xin, et al.
Publicado: (2024)

Reasoning Language Model Inference Serving Unveiled: An Empirical Study
por: Li, Qi, et al.
Publicado: (2025)

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
por: Tang, Zichen, et al.
Publicado: (2024)

Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing
por: Wang, Yuxin, et al.
Publicado: (2023)

Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
por: Lai, Kunfeng, et al.
Publicado: (2025)

LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
por: Zhang, Longteng, et al.
Publicado: (2023)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
por: Pan, Bowen, et al.
Publicado: (2024)

Rethinking Deep Research from the Perspective of Web Content Distribution Matching
por: Yu, Zixuan, et al.
Publicado: (2026)

Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
por: Chu, Kexin, et al.
Publicado: (2025)

Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
por: Dong, Peijie, et al.
Publicado: (2025)

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models
por: Tang, Anke, et al.
Publicado: (2024)

Scalable Training of Mixture-of-Experts Models with Megatron Core
por: Yan, Zijie, et al.
Publicado: (2026)

Should We Really Edit Language Models? On the Evaluation of Edited Language Models
por: Li, Qi, et al.
Publicado: (2024)

Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
por: Nikolic, Strahinja, et al.
Publicado: (2025)

MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
por: Cai, Weilin, et al.
Publicado: (2024)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
por: Qiu, Zihan, et al.
Publicado: (2025)

CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
por: Pham, Quang, et al.
Publicado: (2024)

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
por: Panda, Ashwinee, et al.
Publicado: (2025)

Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
por: Zhang, Longteng, et al.
Publicado: (2026)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
por: Belenki, Lior, et al.
Publicado: (2025)

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
por: Gu, Naibin, et al.
Publicado: (2025)

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs
por: Liu, Enshu, et al.
Publicado: (2024)

Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layers
por: Pavlitska, Svetlana, et al.
Publicado: (2025)

Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMs
por: Jiang, Yukun, et al.
Publicado: (2026)

FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusion
por: Tang, Zhenheng, et al.
Publicado: (2024)

Expert Merging in Sparse Mixture of Experts with Nash Bargaining
por: Nguyen, Dung V., et al.
Publicado: (2025)

MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators
por: Wan, Cheng, et al.
Publicado: (2025)

RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Merging
por: He, Xin, et al.
Publicado: (2025)

Discussion: Effective and Interpretable Outcome Prediction by Training Sparse Mixtures of Linear Experts
por: Folino, Francesco, et al.
Publicado: (2024)

On the Spectral Flattening of Quantized Embeddings
por: Huang, Junlin, et al.
Publicado: (2026)

Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structures
por: Nguyen-Nhat, Minh-Khoi, et al.
Publicado: (2025)