Copertina

Convergence Rate Analysis of the AdamW-Style Shampoo: Unifying One-Sided and Two-Sided Preconditioning

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Li, Huan, Dong, Yiming, Lin, Zhouchen
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Optimization and Control Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

On the $O(\frac{\sqrt{d}}{K^{1/4}})$ Convergence Rate of AdamW Measured by $\ell_1$ Norm
di: Li, Huan, et al.
Pubblicazione: (2025)

Convergence Rate Analysis of LION
di: Dong, Yiming, et al.
Pubblicazione: (2024)

Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization
di: Xie, Shuo, et al.
Pubblicazione: (2024)

HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization
di: Huang, Feihu, et al.
Pubblicazione: (2026)

Optimizer-Induced Mode Connectivity: From AdamW to Muon
di: Zhang, Fangzhao, et al.
Pubblicazione: (2026)

Convergence Rate Analysis of SOAP with Arbitrary Orthogonal Projection Matrices
di: Li, Huan, et al.
Pubblicazione: (2026)

Accelerated Gradient Tracking over Time-varying Graphs for Decentralized Optimization
di: Li, Huan, et al.
Pubblicazione: (2021)

On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm
di: Li, Huan, et al.
Pubblicazione: (2024)

Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
di: Yu, Yaxin, et al.
Pubblicazione: (2026)

One-Sided Matrix Completion from Ultra-Sparse Samples
di: Zhang, Hongyang R., et al.
Pubblicazione: (2026)

Towards Quantifying the Preconditioning Effect of Adam
di: Das, Rudrajit, et al.
Pubblicazione: (2024)

A Unifying View of Anchoring via Operator-Side Tikhonov Regularization
di: Chen, Zihao
Pubblicazione: (2026)

SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration
di: Kovalev, Dmitry
Pubblicazione: (2025)

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
di: Hong, Yusu, et al.
Pubblicazione: (2024)

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
di: Xie, Xingyu, et al.
Pubblicazione: (2022)

A New Perspective on Shampoo's Preconditioner
di: Morwani, Depen, et al.
Pubblicazione: (2024)

Learning-Based Pricing and Matching for Two-Sided Queues
di: Yang, Zixian, et al.
Pubblicazione: (2024)

Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
di: Oikonomidis, Konstantinos, et al.
Pubblicazione: (2026)

Beyond likelihood ratio bias: Nested multi-time-scale stochastic approximation for likelihood-free parameter estimation
di: Li, Zehao, et al.
Pubblicazione: (2024)

Adam Converges Without Any Modification On Update Rules
di: Zhang, Yushun, et al.
Pubblicazione: (2026)

Convergence rates for the Adam optimizer
di: Dereich, Steffen, et al.
Pubblicazione: (2024)

Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization
di: Yu, Yaxin, et al.
Pubblicazione: (2026)

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD
di: Jin, Ruinan, et al.
Pubblicazione: (2024)

Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees
di: Xiao, Nachuan, et al.
Pubblicazione: (2023)

Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
di: Vaswani, Sharan, et al.
Pubblicazione: (2026)

UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization
di: Jiang, Yiming, et al.
Pubblicazione: (2023)

Unified Convergence Analysis for Score-Based Diffusion Models with Deterministic Samplers
di: Li, Runjia, et al.
Pubblicazione: (2024)

On the Convergence of Adam-Type Algorithm for Bilevel Optimization under Unbounded Smoothness
di: Gong, Xiaochuan, et al.
Pubblicazione: (2025)

Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate
di: Lin, Yifan, et al.
Pubblicazione: (2024)

On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
di: Wang, Bohan, et al.
Pubblicazione: (2024)

Theoretical Analysis on how Learning Rate Warmup Accelerates Convergence
di: Liu, Yuxing, et al.
Pubblicazione: (2025)

Unified Convergence Analysis for Adaptive Optimization with Moving Average Estimator
di: Guo, Zhishuai, et al.
Pubblicazione: (2021)

Convergence Rate in Nonlinear Two-Time-Scale Stochastic Approximation with State (Time)-Dependence
di: Chen, Zixi, et al.
Pubblicazione: (2025)

Preconditioned Norms: A Unified Framework for Steepest Descent, Quasi-Newton and Adaptive Methods
di: Veprikov, Andrey, et al.
Pubblicazione: (2025)

PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
di: Lau, Tim Tsz-Kit, et al.
Pubblicazione: (2025)

A Short and Unified Convergence Analysis of the SAG, SAGA, and IAG Algorithms
di: Zhu, Feng, et al.
Pubblicazione: (2026)

Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance
di: Zhang, Qi, et al.
Pubblicazione: (2024)

A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings
di: Mazumder, Alokendu, et al.
Pubblicazione: (2023)

Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates
di: Kovalev, Dmitry, et al.
Pubblicazione: (2025)

MAP Estimation with Denoisers: Convergence Rates and Guarantees
di: Pesme, Scott, et al.
Pubblicazione: (2025)