Image de couverture de livre

Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Pan, Xinglin, Lin, Wenxiang, Shi, Shaohuai, Chu, Xiaowen, Sun, Weinong, Li, Bo
Format:	Preprint
Publié:	2024
Sujets:	Distributed, Parallel, and Cluster Computing Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
par: Lin, Wenxiang, et autres
Publié: (2025)

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
par: Lin, Wenxiang, et autres
Publié: (2026)

Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
par: Pan, Xinglin, et autres
Publié: (2025)

DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
par: Tang, Zhenheng, et autres
Publié: (2025)

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
par: Tang, Zichen, et autres
Publié: (2024)

FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
par: Tang, Zhenheng, et autres
Publié: (2024)

FedImpro: Measuring and Improving Client Update in Federated Learning
par: Tang, Zhenheng, et autres
Publié: (2024)

Fault-Tolerant Hybrid-Parallel Training at Scale with Reliable and Efficient In-memory Checkpointing
par: Wang, Yuxin, et autres
Publié: (2023)

AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
par: Lin, Wenxiang, et autres
Publié: (2026)

Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
par: Wang, Yujie, et autres
Publié: (2024)

Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design
par: Xue, Chunyu, et autres
Publié: (2024)

Accelerating AIGC Services with Latent Action Diffusion Scheduling in Edge Networks
par: Xu, Changfu, et autres
Publié: (2024)

Semantic-Aware Scheduling for GPU Clusters with Large Language Models
par: Wang, Zerui, et autres
Publié: (2025)

Optimizing Large Model Training through Overlapped Activation Recomputation
par: Chen, Ping, et autres
Publié: (2024)

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
par: Fan, Ruibo, et autres
Publié: (2026)

MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
par: Cai, Weilin, et autres
Publié: (2024)

Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training
par: Zhang, Xin, et autres
Publié: (2025)

Understanding Stragglers in Large Model Training Using What-if Analysis
par: Lin, Jinkun, et autres
Publié: (2025)

SparDL: Distributed Deep Learning Training with Efficient Sparse Communication
par: Zhao, Minjun, et autres
Publié: (2023)

Minder: Faulty Machine Detection for Large-scale Distributed Model Training
par: Deng, Yangtao, et autres
Publié: (2024)

Efficient Parallelization Layouts for Large-Scale Distributed Model Training
par: Hagemann, Johannes, et autres
Publié: (2023)

DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training
par: Qiang, Xinwei, et autres
Publié: (2026)

EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models
par: Tan, Zheyue, et autres
Publié: (2025)

Designing Large Foundation Models for Efficient Training and Inference: A Survey
par: Liu, Dong, et autres
Publié: (2024)

AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training
par: Bai, Huawei, et autres
Publié: (2025)

Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaboration
par: Wu, Zhiyuan, et autres
Publié: (2023)

MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
par: Jin, Chao, et autres
Publié: (2025)

Efficient Long-context Language Model Training by Core Attention Disaggregation
par: Zhuang, Yonghao, et autres
Publié: (2025)

Timeliness-Oriented Scheduling and Resource Allocation in Multi-Region Collaborative Perception
par: Zhu, Mengmeng, et autres
Publié: (2026)

Llumnix: Dynamic Scheduling for Large Language Model Serving
par: Sun, Biao, et autres
Publié: (2024)

Preble: Efficient Distributed Prompt Scheduling for LLM Serving
par: Srivatsa, Vikranth, et autres
Publié: (2024)

Efficient Resource Scheduling for Distributed Infrastructures Using Negotiation Capabilities
par: Chu, Junjie, et autres
Publié: (2024)

HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference
par: Zhong, Shuzhang, et autres
Publié: (2025)

Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation
par: Chen, Shaoyuan, et autres
Publié: (2024)

DPQuant: Efficient and Differentially-Private Model Training via Dynamic Quantization Scheduling
par: Gao, Yubo, et autres
Publié: (2025)

PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference
par: Zhang, Zeyu, et autres
Publié: (2024)

Reducing Energy Bloat in Large Model Training
par: Chung, Jae-Won, et autres
Publié: (2023)

Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networks
par: Waleffe, Roger, et autres
Publié: (2025)

A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
par: Barley, Daniel, et autres
Publié: (2026)

Communication-Efficient Device Scheduling for Federated Learning Using Lyapunov Optimization
par: Perazzone, Jake B., et autres
Publié: (2025)