Copertina

Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Lau, Tim Tsz-Kit, Li, Weijian, Xu, Chenwei, Liu, Han, Kolar, Mladen
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Optimization and Control
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2024)

AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2024)

PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2025)

Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2026)

Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
di: Yukhimchuk, Alexander, et al.
Pubblicazione: (2026)

On the Role of Batch Size in Stochastic Conditional Gradient Methods
di: Islamov, Rustem, et al.
Pubblicazione: (2026)

AdaBatchGrad: Combining Adaptive Batch Size and Adaptive Step Size
di: Ostroukhov, Petr, et al.
Pubblicazione: (2024)

Communication-Efficient Gradient Descent-Accent Methods for Distributed Variational Inequalities: Unified Analysis and Local Updates
di: Zhang, Siqi, et al.
Pubblicazione: (2023)

Faster Convergence of Riemannian Stochastic Gradient Descent with Increasing Batch Size
di: Oowada, Kanata, et al.
Pubblicazione: (2025)

Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent
di: Umeda, Hikaru, et al.
Pubblicazione: (2024)

Adaptive Batch Size and Learning Rate Scheduler for Stochastic Gradient Descent Based on Minimization of Stochastic First-order Oracle Complexity
di: Umeda, Hikaru, et al.
Pubblicazione: (2025)

Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
di: Choudhury, Sayantan, et al.
Pubblicazione: (2026)

Fully Stochastic Trust-Region Sequential Quadratic Programming for Equality-Constrained Optimization Problems
di: Fang, Yuchen, et al.
Pubblicazione: (2022)

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent
di: Köhne, Frederik, et al.
Pubblicazione: (2023)

Relationship between Batch Size and Number of Steps Needed for Nonconvex Optimization of Stochastic Gradient Descent using Armijo Line Search
di: Tsukada, Yuki, et al.
Pubblicazione: (2023)

Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
di: Zhang, Dake, et al.
Pubblicazione: (2024)

Trust-Region Sequential Quadratic Programming for Stochastic Optimization with Random Models
di: Fang, Yuchen, et al.
Pubblicazione: (2024)

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
di: Tao, Yuanzhe, et al.
Pubblicazione: (2024)

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
di: Zhou, Dongruo, et al.
Pubblicazione: (2018)

Directional Smoothness and Gradient Methods: Convergence and Adaptivity
di: Mishkin, Aaron, et al.
Pubblicazione: (2024)

A Weighted Gradient Tracking Privacy-Preserving Method for Distributed Optimization
di: Xie, Furan, et al.
Pubblicazione: (2025)

Randomized Feasibility Methods for Constrained Optimization with Adaptive Step Sizes
di: Chakraborty, Abhishek, et al.
Pubblicazione: (2026)

Convergence of Distributed Adaptive Optimization with Local Updates
di: Cheng, Ziheng, et al.
Pubblicazione: (2024)

A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting
di: Tyurin, Alexander, et al.
Pubblicazione: (2022)

Optimal Growth Schedules for Batch Size and Learning Rate in SGD that Reduce SFO Complexity
di: Umeda, Hikaru, et al.
Pubblicazione: (2025)

Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch
di: Wang, Weizhen, et al.
Pubblicazione: (2025)

On Penalty-based Bilevel Gradient Descent Method
di: Shen, Han, et al.
Pubblicazione: (2023)

Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGrad
di: Liu, Zijian
Pubblicazione: (2026)

Nesterov Finds GRAAL: Optimal and Adaptive Gradient Method for Convex Optimization
di: Borodich, Ekaterina, et al.
Pubblicazione: (2025)

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes
di: Orvieto, Antonio, et al.
Pubblicazione: (2024)

Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning
di: Bylinkin, Dmitry, et al.
Pubblicazione: (2024)

Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
di: Harada, Hinata, et al.
Pubblicazione: (2024)

Both Asymptotic and Non-Asymptotic Convergence of Quasi-Hyperbolic Momentum using Increasing Batch Size
di: Imaizumi, Kento, et al.
Pubblicazione: (2025)

Online Linear Programming with Batching
di: Xu, Haoran, et al.
Pubblicazione: (2024)

Batched First-Order Methods for Parallel LP Solving in MIP
di: Blin, Nicolas, et al.
Pubblicazione: (2026)

Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
di: Suh, Min-Kook, et al.
Pubblicazione: (2024)

Communication-Efficient Stochastic Distributed Learning
di: Ren, Xiaoxing, et al.
Pubblicazione: (2025)

Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
di: Wang, Jinbo, et al.
Pubblicazione: (2026)

ASGO: Adaptive Structured Gradient Optimization
di: An, Kang, et al.
Pubblicazione: (2025)

Elementary Analysis of Policy Gradient Methods
di: Liu, Jiacai, et al.
Pubblicazione: (2024)