Cover Image

DiLoCo: Distributed Low-Communication Training of Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Douillard, Arthur, Feng, Qixuan, Rusu, Andrei A., Chhaparia, Rachita, Donchev, Yani, Kuncoro, Adhiguna, Ranzato, Marc'Aurelio, Szlam, Arthur, Shen, Jiajun
Format:	Preprint
Published:	2023
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DiPaCo: Distributed Path Composition
by: Douillard, Arthur, et al.
Published: (2024)

Asynchronous Local-SGD Training for Language Modeling
by: Liu, Bo, et al.
Published: (2024)

Context Training with Active Information Seeking
by: Huang, Zeyu, et al.
Published: (2026)

Decoupled DiLoCo for Resilient Distributed Pre-training
by: Douillard, Arthur, et al.
Published: (2026)

Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
by: Douillard, Arthur, et al.
Published: (2025)

Eager Updates For Overlapped Communication and Computation in DiLoCo
by: Kale, Satyen, et al.
Published: (2025)

Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
by: Charles, Zachary, et al.
Published: (2025)

Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs
by: Defazio, Aaron, et al.
Published: (2025)

What happens when nanochat meets DiLoCo?
by: Acker, Alexander, et al.
Published: (2025)

MuLoCo: Muon is a practical inner optimizer for DiLoCo
by: Thérien, Benjamin, et al.
Published: (2025)

Latent Space Communication via K-V Cache Alignment
by: Dery, Lucio M., et al.
Published: (2026)

Cosine-Gated Adam-Decay: Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo
by: Shah, Vatsal, et al.
Published: (2026)

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
by: Jaghouar, Sami, et al.
Published: (2024)

DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster
by: Qi, Ji, et al.
Published: (2025)

Biserica ortodoxă din Republica Moldova în contextul geopoliticii europene: între Russkii Mir şi Biserica Neamului
by: Rusu, Andrei
Published: (2024)

Evaluating an Effective Ransomware Infection Vector in Low Earth Orbit Satellites
by: Donchev, Marin, et al.
Published: (2024)

Deliberation in Latent Space via Differentiable Cache Augmentation
by: Liu, Luyang, et al.
Published: (2024)

Il quinto secolo. Studi di filosofia antica in onore di livio rossetti. A cura di S. Giombini e F. Marcacci, Perugia, 2010
by: Sofia Ranzato
Published: (2011)

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
by: Shafiullah, Nur Muhammad Mahi, et al.
Published: (2022)

LoCo: Low-Bit Communication Adaptor for Large-scale Model Training
by: Xie, Xingyu, et al.
Published: (2024)

Distribution of integer points on determinant surfaces and a mod‐ p analogue
by: Satadal Ganguly, et al.
Published: (2026)

Low-Perplexity LLM-Generated Sequences and Where To Find Them
by: Wuhrmann, Arthur, et al.
Published: (2025)

LoCoDL: Communication-Efficient Distributed Learning with Local Training and Compression
by: Condat, Laurent, et al.
Published: (2024)

Distribution of integer points on determinant surfaces and a $\text{mod-}p$ analogue
by: Ganguly, Satadal, et al.
Published: (2025)

Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
by: Khaled, Ahmed, et al.
Published: (2025)

NoLoCo: No-all-reduce Low Communication Training Method for Large Models
by: Kolehmainen, Jari, et al.
Published: (2025)

Automating the Attendance Tracking Process in Educational Institutions - Figure 3
by: Daniel Damyanov, et al.
Published: (2025)

Counting $2\times2$ integer matrices with fixed trace and determinant
by: Guria, Rachita
Published: (2024)

An asymptotic formula with power-saving error term for counting prime solutions to a binary additive problem
by: Guria, Rachita
Published: (2024)

Private LoRA Fine-tuning of Open-Source LLMs with Homomorphic Encryption
by: Frery, Jordan, et al.
Published: (2025)

Caesalpinioideae (leguminosae) lenhosas na Estação Ambiental de Volta Grande, Minas Gerais, Brasil
by: Fabiana Luiza Ranzato Filardi
Published: (2009)

Customer-Service Training in Academic Libraries.
by: Arthur, Gwen
Published: (1994)

Learning Explicitly Conditioned Sparsifying Transforms
by: Pătraşcu, Andrei, et al.
Published: (2024)

Community Detection with the Canonical Ensemble
by: Arthur, Rudy
Published: (2026)

Communication and Cooperation in Public Libraries
by: Meyers, Arthur
Published: (1978)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
by: Yang, Yifan, et al.
Published: (2024)

A Systematic Literature Review of Fleksibilitas Dan Budaya Organisasi Penting Bagi UKM Di Pasar Yang Kompetitif
by: Yani Dwi Restanti, et al.
Published: (2025)

Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff
by: Jacot, Arthur
Published: (2023)

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
by: Jovanović, Andrej, et al.
Published: (2026)

Training and Gaining School Library Volunteers.
by: Hoagland, Mary Arthur
Published: (1984)