Cover Image

MuLoCo: Muon is a practical inner optimizer for DiLoCo

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Thérien, Benjamin, Huang, Xiaolong, Defazio, Aaron, Rish, Irina, Belilovsky, Eugene
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs
by: Defazio, Aaron, et al.
Published: (2025)

Communication Efficient LLM Pre-training with SparseLoCo
by: Sarfi, Amir, et al.
Published: (2025)

DiLoCo: Distributed Low-Communication Training of Language Models
by: Douillard, Arthur, et al.
Published: (2023)

What happens when nanochat meets DiLoCo?
by: Acker, Alexander, et al.
Published: (2025)

$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
by: Thérien, Benjamin, et al.
Published: (2024)

Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training
by: Singh, Vaibhav, et al.
Published: (2025)

Cosine-Gated Adam-Decay: Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo
by: Shah, Vatsal, et al.
Published: (2026)

Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
by: Legate, Gwen, et al.
Published: (2025)

PyLO: Towards Accessible Learned Optimizers in PyTorch
by: Janson, Paul, et al.
Published: (2025)

Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
by: Charles, Zachary, et al.
Published: (2025)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
by: Ibrahim, Adam, et al.
Published: (2024)

Eager Updates For Overlapped Communication and Computation in DiLoCo
by: Kale, Satyen, et al.
Published: (2025)

Decoupled DiLoCo for Resilient Distributed Pre-training
by: Douillard, Arthur, et al.
Published: (2026)

Meta-learning Optimizers for Communication-Efficient Learning
by: Joseph, Charles-Étienne, et al.
Published: (2023)

Continual Pre-training of MoEs: How robust is your router?
by: Thérien, Benjamin, et al.
Published: (2025)

Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
by: Douillard, Arthur, et al.
Published: (2025)

Why Gradients Rapidly Increase Near the End of Training
by: Defazio, Aaron
Published: (2025)

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
by: Defazio, Aaron
Published: (2026)

FairDropout: Using Example-Tied Dropout to Enhance Generalization of Minority Groups
by: Nanfack, Geraldin, et al.
Published: (2025)

Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL
by: Miahi, Erfan, et al.
Published: (2026)

OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
by: Jaghouar, Sami, et al.
Published: (2024)

LoCoCo: Dropping In Convolutions for Long Context Compression
by: Cai, Ruisi, et al.
Published: (2024)

Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
by: Davari, MohammadReza, et al.
Published: (2023)

Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
by: Lidin, Joel, et al.
Published: (2026)

DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster
by: Qi, Ji, et al.
Published: (2025)

MuCon: Clipped Muon Updates for LLM Training
by: Yi, Albert
Published: (2026)

Prodigy: An Expeditiously Adaptive Parameter-Free Learner
by: Mishchenko, Konstantin, et al.
Published: (2023)

When Data Falls Short: Grokking Below the Critical Threshold
by: Singh, Vaibhav, et al.
Published: (2025)

Stabilizing Native Low-Rank LLM Pretraining
by: Janson, Paul, et al.
Published: (2026)

Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
by: Singh, Vaibhav, et al.
Published: (2024)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
by: Abbes, Istabrak, et al.
Published: (2025)

Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization
by: Kang, Changmin, et al.
Published: (2026)

Celo2: Towards Learned Optimization Free Lunch
by: Moudgil, Abhinav, et al.
Published: (2026)

Efficient Refusal Ablation in LLM through Optimal Transport
by: Nanfack, Geraldin, et al.
Published: (2026)

Celo: Training Versatile Learned Optimizers on a Compute Diet
by: Moudgil, Abhinav, et al.
Published: (2025)

CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations
by: Berman, Jules, et al.
Published: (2024)

Knowledge Distillation for Federated Learning: a Practical Guide
by: Mora, Alessio, et al.
Published: (2022)

NoLoCo: No-all-reduce Low Communication Training Method for Large Models
by: Kolehmainen, Jari, et al.
Published: (2025)

Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting
by: Hameed, Humza Wajid, et al.
Published: (2024)

CoLoRSMamba: Conditional LoRA-Steered Mamba for Supervised Multimodal Violence Detection
by: Senadeera, Damith Chamalke, et al.
Published: (2026)