Cover Image

The Limits and Potentials of Local SGD for Distributed Heterogeneous Learning with Intermittent Communication

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Patel, Kumar Kshitij, Glasgow, Margalit, Zindari, Ali, Wang, Lingxiao, Stich, Sebastian U., Cheng, Ziheng, Joshi, Nirmit, Srebro, Nathan
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Distributed, Parallel, and Cluster Computing Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysis
by: Luo, Ruichen, et al.
Published: (2025)

Efficient Gradient Methods for Distributed Saddle Problems
by: Luo, Ruichen, et al.
Published: (2026)

dSTAR: Straggler Tolerant and Byzantine Resilient Distributed SGD
by: Yan, Jiahe, et al.
Published: (2024)

PowerTrip: Exploiting Federated Heterogeneous Datacenter Power for Distributed ML Training
by: Mehboob, Talha, et al.
Published: (2025)

Mesh-Attention: A New Communication-Efficient Distributed Attention with Improved Data Locality
by: Chen, Sirui, et al.
Published: (2025)

Distributed Computation with Local Advice
by: Balliu, Alkida, et al.
Published: (2024)

EdgeSphere: A Three-Tier Architecture for Cognitive Edge Computing
by: Makaya, Christian, et al.
Published: (2024)

A Post-Quantum Lower Bound for the Distributed Lovász Local Lemma
by: Brandt, Sebastian, et al.
Published: (2025)

EDiT: A Local-SGD-Based Efficient Distributed Training Method for Large Language Models
by: Cheng, Jialiang, et al.
Published: (2024)

Truncated Non-Uniform Quantization for Distributed SGD
by: Yan, Guangfeng, et al.
Published: (2024)

LOSCAR-SGD: Local SGD with Communication-Computation Overlap and Delay-Corrected Sparse Model Averaging
by: Maziane, Yassine, et al.
Published: (2026)

CXL Shared Memory Programming: Barely Distributed and Almost Persistent
by: Xu, Yi, et al.
Published: (2024)

CheckMate: LLM-Powered Approximate Intermittent Computing
by: Sayyid-Ali, Abdur-Rahman Ibrahim, et al.
Published: (2024)

MPI-Q: A Message Communication Library for Large-Scale Classical-Quantum Heterogeneous Hybrid Distributed Computing
by: Wang, Feng, et al.
Published: (2026)

On The Reproducibility Limitations of RAG Systems
by: Wang, Baiqiang, et al.
Published: (2025)

Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity
by: Mahran, Ammar, et al.
Published: (2026)

Distributed Algorithms for Potential Problems
by: Balliu, Alkida, et al.
Published: (2025)

Federated Inference for Heterogeneous LLM Communication and Collaboration
by: Chen, Zihan, et al.
Published: (2026)

Personalized Federated Learning on Data with Dynamic Heterogeneity under Limited Storage
by: Tan, Sixing, et al.
Published: (2024)

HiCR, an Abstract Model for Distributed Heterogeneous Programming
by: Martin, Sergio Miguel, et al.
Published: (2025)

Addressing Variable Heterogeneity in Distributed Multimodal Training with Entrain
by: Jang, Insu, et al.
Published: (2026)

Toward Heterogeneous, Distributed, and Energy-Efficient Computing with SYCL
by: Cosenza, Biagio, et al.
Published: (2025)

Heta: Distributed Training of Heterogeneous Graph Neural Networks
by: Zhong, Yuchen, et al.
Published: (2024)

RainCloud: Decentralized Coordination and Communication in Heterogeneous IoT Swarms
by: Loisel, Filip, et al.
Published: (2024)

MindFlayer SGD: Efficient Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times
by: Maranjyan, Artavazd, et al.
Published: (2024)

Error-resilient Distributed Local Verification
by: Garncarek, Paweł, et al.
Published: (2026)

DiOMP-Offloading: Toward Portable Distributed Heterogeneous OpenMP
by: Shan, Baodi, et al.
Published: (2025)

DFPL: Decentralized Federated Prototype Learning Across Heterogeneous Data Distributions
by: Zhang, Hongliang, et al.
Published: (2025)

Poplar: Efficient Scaling of Distributed DNN Training on Heterogeneous GPU Clusters
by: Zhang, WenZheng, et al.
Published: (2024)

A Comparative Analysis of Distributed Training Strategies for GPT-2
by: Patwardhan, Ishan, et al.
Published: (2024)

Polylog-Competitive Deterministic Local Routing and Scheduling
by: Haeupler, Bernhard, et al.
Published: (2024)

Birch SGD: A Tree Graph Framework for Local and Asynchronous SGD Methods
by: Tyurin, Alexander, et al.
Published: (2025)

Shared Randomness Helps with Local Distributed Problems
by: Balliu, Alkida, et al.
Published: (2024)

D-Rex: Heterogeneity-Aware Reliability Framework and Adaptive Algorithms for Distributed Storage
by: Gonthier, Maxime, et al.
Published: (2025)

Sailor: Automating Distributed Training over Dynamic, Heterogeneous, and Geo-distributed Clusters
by: Strati, Foteini, et al.
Published: (2025)

Modeling the Potential of Message-Free Communication via CXL.mem
by: Vanecek, Stepan, et al.
Published: (2025)

Fault-Tolerant Decentralized Distributed Asynchronous Federated Learning with Adaptive Termination Detection
by: Akkinepally, Phani Sahasra, et al.
Published: (2025)

ColonyOS -- A Meta-Operating System for Distributed Computing Across Heterogeneous Platform
by: Kristiansson, Johan
Published: (2024)

Ordered Momentum for Asynchronous SGD
by: Shi, Chang-Wei, et al.
Published: (2024)

Overcoming Latency-bound Limitations of Distributed Graph Algorithms using the HPX Runtime System
by: Mohammadiporshokooh, Karame, et al.
Published: (2026)