DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910972883501056 |
|---|---|
| author | Iacob, Alex Sani, Lorenzo Safaryan, Mher Giampouras, Paris Horváth, Samuel Jovanovic, Andrej Kurmanji, Meghdad Aleksandrov, Preslav Shen, William F. Qiu, Xinchi Lane, Nicholas D. |
| author_facet | Iacob, Alex Sani, Lorenzo Safaryan, Mher Giampouras, Paris Horváth, Samuel Jovanovic, Andrej Kurmanji, Meghdad Aleksandrov, Preslav Shen, William F. Qiu, Xinchi Lane, Nicholas D. |
| contents | Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize only model parameters and cannot be trivially applied to adaptive optimizers due to additional optimizer states. Current approaches extending Local SGD either lack convergence guarantees or require synchronizing all optimizer states, tripling communication costs. We propose Desynced Low Communication Adaptive Optimizers (DES-LOC), a family of optimizers assigning independent synchronization periods to parameters and momenta, enabling lower communication costs while preserving convergence. Through extensive experiments on language models of up to 1.7B, we show that DES-LOC can communicate 170x less than DDP and 2x less than the previous state-of-the-art Local ADAM. Furthermore, unlike previous heuristic approaches, DES-LOC is suited for practical training scenarios prone to system failures. DES-LOC offers a scalable, bandwidth-efficient, and fault-tolerant solution for foundation model training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22549 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models Iacob, Alex Sani, Lorenzo Safaryan, Mher Giampouras, Paris Horváth, Samuel Jovanovic, Andrej Kurmanji, Meghdad Aleksandrov, Preslav Shen, William F. Qiu, Xinchi Lane, Nicholas D. Machine Learning Scaling foundation model training with Distributed Data Parallel (DDP) methods is bandwidth-limited. Existing infrequent communication methods like Local SGD were designed to synchronize only model parameters and cannot be trivially applied to adaptive optimizers due to additional optimizer states. Current approaches extending Local SGD either lack convergence guarantees or require synchronizing all optimizer states, tripling communication costs. We propose Desynced Low Communication Adaptive Optimizers (DES-LOC), a family of optimizers assigning independent synchronization periods to parameters and momenta, enabling lower communication costs while preserving convergence. Through extensive experiments on language models of up to 1.7B, we show that DES-LOC can communicate 170x less than DDP and 2x less than the previous state-of-the-art Local ADAM. Furthermore, unlike previous heuristic approaches, DES-LOC is suited for practical training scenarios prone to system failures. DES-LOC offers a scalable, bandwidth-efficient, and fault-tolerant solution for foundation model training. |
| title | DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.22549 |