Image de couverture de livre

Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Defazio, Aaron, Mishchenko, Konstantin, Raman, Parameswaran, Shi, Hao-Jun Michael, Xiao, Lin
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Prodigy: An Expeditiously Adaptive Parameter-Free Learner
par: Mishchenko, Konstantin, et autres
Publié: (2023)

MuLoCo: Muon is a practical inner optimizer for DiLoCo
par: Thérien, Benjamin, et autres
Publié: (2025)

Optimal Linear Decay Learning Rate Schedules and Further Refinements
par: Defazio, Aaron, et autres
Publié: (2023)

What happens when nanochat meets DiLoCo?
par: Acker, Alexander, et autres
Publié: (2025)

Why Gradients Rapidly Increase Near the End of Training
par: Defazio, Aaron
Publié: (2025)

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
par: Defazio, Aaron
Publié: (2026)

The Road Less Scheduled
par: Defazio, Aaron, et autres
Publié: (2024)

DiLoCo: Distributed Low-Communication Training of Language Models
par: Douillard, Arthur, et autres
Publié: (2023)

Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner
par: Eschenhagen, Runa, et autres
Publié: (2025)

DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster
par: Qi, Ji, et autres
Publié: (2025)

Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
par: Naganuma, Hiroki, et autres
Publié: (2026)

Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo
par: Charles, Zachary, et autres
Publié: (2025)

Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithm
par: Xu, Yang, et autres
Publié: (2025)

Learning Graph Node Embeddings by Smooth Pair Sampling
par: Kutzkov, Konstantin
Publié: (2025)

CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation
par: Mei, Kangfu, et autres
Publié: (2023)

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
par: Xiao, Guangxuan, et autres
Publié: (2022)

Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
par: Gautam, Tanmay, et autres
Publié: (2024)

Cosine-Gated Adam-Decay: Drop-In Staleness-Aware Outer Optimization for Decoupled DiLoCo
par: Shah, Vatsal, et autres
Publié: (2026)

DiFFPO: Training Diffusion LLMs to Reason Fast and Furious via Reinforcement Learning
par: Zhao, Hanyang, et autres
Publié: (2025)

Train Faster, Perform Better: Modular Adaptive Training in Over-Parameterized Models
par: Shi, Yubin, et autres
Publié: (2024)

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm
par: Bai, Qinbo, et autres
Publié: (2024)

Conda: Column-Normalized Adam for Training Large Language Models Faster
par: Wang, Junjie, et autres
Publié: (2025)

Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs
par: Li, Qingyuan, et autres
Publié: (2024)

Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
par: Wang, Xu, et autres
Publié: (2025)

Federated Domain Generalization with Label Smoothing and Balanced Decentralized Training
par: Soltany, Milad, et autres
Publié: (2024)

A Theory of Training Profit-Optimal LLMs
par: Hao, Sophie, et autres
Publié: (2026)

Eager Updates For Overlapped Communication and Computation in DiLoCo
par: Kale, Satyen, et autres
Publié: (2025)

Decoupled DiLoCo for Resilient Distributed Pre-training
par: Douillard, Arthur, et autres
Publié: (2026)

Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs
par: Anjum, Afia, et autres
Publié: (2024)

LoRe: Personalizing LLMs via Low-Rank Reward Modeling
par: Bose, Avinandan, et autres
Publié: (2025)

Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
par: Shen, Xuan, et autres
Publié: (2023)

Towards Faster Training of Diffusion Models: An Inspiration of A Consistency Phenomenon
par: Xu, Tianshuo, et autres
Publié: (2024)

Lagrangian Index Policy for Restless Bandits with Average Reward
par: Avrachenkov, Konstantin, et autres
Publié: (2024)

Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
par: Zhou, Tailin, et autres
Publié: (2023)

Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference
par: Yi, Ke, et autres
Publié: (2024)

Fast and Accurate Probing of In-Training LLMs' Downstream Performances
par: Liu, Zhichen, et autres
Publié: (2026)

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
par: Yan, Peishen, et autres
Publié: (2026)

DiRL: An Efficient Post-Training Framework for Diffusion Language Models
par: Zhu, Ying, et autres
Publié: (2025)

Ortho-Hydra: Orthogonalized Experts for DiT LoRA
par: Ji, Seunghyun
Publié: (2026)

CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
par: Liu, Ziyue, et autres
Publié: (2025)