Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908809854713856 |
|---|---|
| author | Begunov, Grigory Tyurin, Alexander |
| author_facet | Begunov, Grigory Tyurin, Alexander |
| contents | Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous SGD and its robust variant, called $m$-Synchronous SGD, and theoretically show that they are nearly optimal in many heterogeneous computation scenarios, which is somewhat unexpected. We analyze the synchronous methods under random computation times and adversarial partial participation of workers, and prove that their time complexities are optimal in many practical regimes, up to logarithmic factors. While synchronous methods are not universal solutions and there exist tasks where asynchronous methods may be necessary, we show that they are sufficient for many modern heterogeneous computation scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_03802 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods Begunov, Grigory Tyurin, Alexander Distributed, Parallel, and Cluster Computing Artificial Intelligence Numerical Analysis Optimization and Control Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous SGD and its robust variant, called $m$-Synchronous SGD, and theoretically show that they are nearly optimal in many heterogeneous computation scenarios, which is somewhat unexpected. We analyze the synchronous methods under random computation times and adversarial partial participation of workers, and prove that their time complexities are optimal in many practical regimes, up to logarithmic factors. While synchronous methods are not universal solutions and there exist tasks where asynchronous methods may be necessary, we show that they are sufficient for many modern heterogeneous computation scenarios. |
| title | Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Numerical Analysis Optimization and Control |
| url | https://arxiv.org/abs/2602.03802 |