Buchumschlag

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models

Fuente: arXiv

Gespeichert in:

Bibliographische Detailangaben
Hauptverfasser:	Cheng, Jialiang, Gao, Ning, Yue, Yun, Ye, Zhiling, Jiang, Jiadi, Sha, Jian
Format:	Preprint
Veröffentlicht:	2024
Schlagworte:	Distributed, Parallel, and Cluster Computing Artificial Intelligence
Online-Zugang:	Acceder al recurso
Tags:	Tag hinzufügen Keine Tags, Fügen Sie den ersten Tag hinzu!

Ähnliche Einträge

AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix
von: Yue, Yun, et al.
Veröffentlicht: (2023)

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey
von: Duan, Jiangfei, et al.
Veröffentlicht: (2024)

dSTAR: Straggler Tolerant and Byzantine Resilient Distributed SGD
von: Yan, Jiahe, et al.
Veröffentlicht: (2024)

Towards Energy Efficient Co-Scheduling in HPC
von: Zheng, Zhong, et al.
Veröffentlicht: (2026)

DiffusionPipe: Training Large Diffusion Models with Efficient Pipelines
von: Tian, Ye, et al.
Veröffentlicht: (2024)

Efficient Distributed MLLM Training with Cornstarch
von: Jang, Insu, et al.
Veröffentlicht: (2025)

RapidGNN: Communication Efficient Large-Scale Distributed Training of Graph Neural Networks
von: Niam, Arefin, et al.
Veröffentlicht: (2025)

An Explorative Study on Distributed Computing Techniques in Training and Inference of Large Language Models
von: Hakim, Sheikh Azizul, et al.
Veröffentlicht: (2025)

MTGenRec: An Efficient Distributed Training System for Generative Recommendation Models in Meituan
von: Wang, Yuxiang, et al.
Veröffentlicht: (2025)

DistTrain: Addressing Model and Data Heterogeneity with Disaggregated Training for Multimodal Large Language Models
von: Zhang, Zili, et al.
Veröffentlicht: (2024)

Seq1F1B: Efficient Sequence-Level Pipeline Parallelism for Large Language Model Training
von: Sun, Ao, et al.
Veröffentlicht: (2024)

Cloud-native and Distributed Systems for Efficient and Scalable Large Language Models -- A Research Agenda
von: Xu, Minxian, et al.
Veröffentlicht: (2026)

Birch SGD: A Tree Graph Framework for Local and Asynchronous SGD Methods
von: Tyurin, Alexander, et al.
Veröffentlicht: (2025)

MoLink: Distributed and Efficient Serving Framework for Large Models
von: Jin, Lewei, et al.
Veröffentlicht: (2025)

Galvatron: Automatic Distributed Training for Large Transformer Models
von: Gumaan, Esmail
Veröffentlicht: (2025)

ACE-Sync: An Adaptive Cloud-Edge Synchronization Framework for Communication-Efficient Large-Scale Distributed Model Training
von: Yang, Yi, et al.
Veröffentlicht: (2025)

HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware
von: Yan, Ran, et al.
Veröffentlicht: (2024)

SDSL-Solver: Scalable Distributed Sparse Linear Solvers for Large-Scale Interior Point Methods
von: Yang, Shaofeng, et al.
Veröffentlicht: (2026)

MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training
von: Zhao, Bohan, et al.
Veröffentlicht: (2025)

Truncated Non-Uniform Quantization for Distributed SGD
von: Yan, Guangfeng, et al.
Veröffentlicht: (2024)

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication
von: Patel, Kumar Kshitij, et al.
Veröffentlicht: (2024)

Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models
von: Wang, Wei, et al.
Veröffentlicht: (2024)

BurstEngine: an Efficient Distributed Framework for Training Transformers on Extremely Long Sequences of over 1M Tokens
von: Sun, Ao, et al.
Veröffentlicht: (2025)

DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
von: Wang, Zhixin, et al.
Veröffentlicht: (2025)

Cascadia: An Efficient Cascade Serving System for Large Language Models
von: Jiang, Youhe, et al.
Veröffentlicht: (2025)

Metronome: Efficient Scheduling for Periodic Traffic Jobs with Network and Priority Awareness
von: Jiang, Hao, et al.
Veröffentlicht: (2025)

Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters
von: Zhang, WenZheng, et al.
Veröffentlicht: (2024)

EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models
von: Liu, Han, et al.
Veröffentlicht: (2025)

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding
von: Chen, Qiaoling, et al.
Veröffentlicht: (2024)

A Scalable Recipe on SuperMUC-NG Phase 2: Efficient Large-Scale Training of Language Models
von: Rajgopal, Ajay Navilarekal, et al.
Veröffentlicht: (2026)

Accelerating Distributed MoE Training and Inference with Lina
von: Li, Jiamin, et al.
Veröffentlicht: (2022)

Chameleon: Adaptive Fault Tolerance for Distributed Training via Real-time Policy Selection
von: Zhou, Yuhang, et al.
Veröffentlicht: (2025)

More for Less: Integrating Capability-Predominant and Capacity-Predominant Computing
von: Zheng, Zhong, et al.
Veröffentlicht: (2025)

EcoShift: Performance-Aware Power Management for Power-Constrained Heterogeneous Systems
von: Zheng, Zhong, et al.
Veröffentlicht: (2026)

Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimization
von: Zhu, Zhanda, et al.
Veröffentlicht: (2025)

An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters
von: Zhang, Mingjun, et al.
Veröffentlicht: (2025)

CondenseGraph: Communication-Efficient Distributed GNN Training via On-the-Fly Graph Condensation
von: Zhang, Zizhao, et al.
Veröffentlicht: (2026)

GreenDyGNN: Runtime-Adaptive Energy-Efficient Communication for Distributed GNN Training
von: Niam, Arefin, et al.
Veröffentlicht: (2026)

Bandwidth-Aware and Cost-Efficient Pipeline Parallel Scheduling in Geo-Distributed LLM Training
von: Zhang, Han, et al.
Veröffentlicht: (2026)

Efficient Parallelization Layouts for Large-Scale Distributed Model Training
von: Hagemann, Johannes, et al.
Veröffentlicht: (2023)