Cover Image

Decoupled DiLoCo for Resilient Distributed Pre-training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Douillard, Arthur, Rush, Keith, Donchev, Yani, Charles, Zachary, Fallen, Nova, Dubey, Ayush, Gog, Ionel, Dean, Josef, Woodworth, Blake, Garrett, Zachary, Keating, Nate, Bishop, Jenny, Prior, Henry, Yvinec, Edouard, Szlam, Arthur, Ranzato, Marc'Aurelio, Dean, Jeff
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
by: Charles, Zachary, et al.
Published: (2025)

DiLoCo: Distributed Low-Communication Training of Language Models
by: Douillard, Arthur, et al.
Published: (2023)

Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
by: Douillard, Arthur, et al.
Published: (2025)

DiPaCo: Distributed Path Composition
by: Douillard, Arthur, et al.
Published: (2024)

Eager Updates For Overlapped Communication and Computation in DiLoCo
by: Kale, Satyen, et al.
Published: (2025)

Asynchronous Local-SGD Training for Language Modeling
by: Liu, Bo, et al.
Published: (2024)

Context Training with Active Information Seeking
by: Huang, Zeyu, et al.
Published: (2026)

Federated Automatic Differentiation
by: Rush, Keith, et al.
Published: (2023)

SPOT: Text Source Prediction from Originality Score Thresholding
by: Yvinec, Edouard, et al.
Published: (2024)

DrJAX: Scalable and Differentiable MapReduce Primitives in JAX
by: Rush, Keith, et al.
Published: (2024)

Latent Space Communication via K-V Cache Alignment
by: Dery, Lucio M., et al.
Published: (2026)

Effective description of Taylor dispersion in strongly corrugated channels
by: Alexandre, Arthur, et al.
Published: (2025)

(Table 1) Shipboard sediment descriptions at DSDP Hole 93-603B
by: Dean, Walter E, et al.
Published: (1987)

(Table 15) Communalities and factor loadings used in the Q-mode model for sediments from DSDP Hole 93-603B
by: Dean, Walter E, et al.
Published: (1987)

Inorganic and organic geochemistry and sediment descriptions at DSDP Hole 93-603B
by: Dean, Walter E, et al.
Published: (1987)

Geochemistry of Holocene Black Sea sediments
by: Arthur, Michael A, et al.
Published: (2016)

(Table 4) Inorganic and organic geochemistry of postcruise samples at DSDP Hole 93-603B
by: Dean, Walter E, et al.
Published: (1987)

(Table 9) Calcium carbonate, organic carbon, hydrogen and oxygen index and Q-mode factor at DSDP Hole 93-603B
by: Dean, Walter E, et al.
Published: (1987)

Geochemistry of sediment core KN134-8_Leg1_GC08
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC09
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC20
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC66
by: Arthur, Michael A, et al.
Published: (1998)

(Table 2) Postcruise sediment descriptions at DSDP Hole 93-603B
by: Dean, Walter E, et al.
Published: (1987)

(Table 3) Inorganic and organic geochemistry of shipboard samples at DSDP Hole 93-603B
by: Dean, Walter E, et al.
Published: (1987)

Geochemistry of sediment core KN134-8_Leg1_GC19
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC38
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC71
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC01
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC42
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC59
by: Arthur, Michael A, et al.
Published: (1998)

Geochemistry of sediment core KN134-8_Leg1_GC79
by: Arthur, Michael A, et al.
Published: (1998)

Reinventar el indio después de la Independencia: las representaciones del indígena en el Perú decimonónico (1821-1879)
by: Maud Yvinec
Published: (2013)

Deliberation in Latent Space via Differentiable Cache Augmentation
by: Liu, Luyang, et al.
Published: (2024)

Il quinto secolo. Studi di filosofia antica in onore di livio rossetti. A cura di S. Giombini e F. Marcacci, Perugia, 2010
by: Sofia Ranzato
Published: (2011)

A botanic garden for the nation : the United States Botanic Garden / by Anne-Catherine Fallen ; contributors, William C. Allen, Karen D. Solit [and] the staff and gardeners of the United States Botanic Garden ; project coordinator, Holly H. Shimizu
by: Fallen, Anne-Catherine
Published: (2006)

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
by: Shafiullah, Nur Muhammad Mahi, et al.
Published: (2022)

(Table 1) Organic carbon content and stable carbon isotope ratios, and hydrogen and oxygen index of DSDP Site 27-259
by: Dean, Walter E, et al.
Published: (1986)

(Table 1) Organic carbon content and stable carbon isotope ratios, and hydrogen and oxygen index of DSDP Site 33-317A
by: Dean, Walter E, et al.
Published: (1986)

(Table 1) Organic carbon content and stable carbon isotope ratios, and hydrogen and oxygen index of several land sections
by: Dean, Walter E, et al.
Published: (1986)

(Table 2) Organic carbon content and stable carbon isotope ratios, and hydrogen and oxygen index of sediment core BX-6
by: Dean, Walter E, et al.
Published: (1986)