Imagen de Portada

Convergence Guarantees for RMSProp and Adam in Generalized-smooth Non-convex Optimization with Affine Noise Variance

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Zhang, Qi, Zhou, Yi, Zou, Shaofeng
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Optimization and Control
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Adam-family Methods for Nonsmooth Optimization with Convergence Guarantees
por: Xiao, Nachuan, et al.
Publicado: (2023)

MGDA Converges under Generalized Smoothness, Provably
por: Zhang, Qi, et al.
Publicado: (2024)

Modeling AdaGrad, RMSProp, and Adam with Integro-Differential Equations
por: Heredia, Carlos
Publicado: (2024)

Random Scaling and Momentum for Non-smooth Non-convex Optimization
por: Zhang, Qinzi, et al.
Publicado: (2024)

Decentralized Non-convex Stochastic Optimization with Heterogeneous Variance
por: Chen, Hongxu, et al.
Publicado: (2026)

Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversion
por: Cutkosky, Ashok, et al.
Publicado: (2023)

Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case
por: He, Meixuan, et al.
Publicado: (2023)

Convergence of Spectral Descent for Non-smooth Optimization
por: Yang, Yixuan, et al.
Publicado: (2026)

Adaptive Gradient Normalization and Independent Sampling for (Stochastic) Generalized-Smooth Optimization
por: Yang, Yufeng, et al.
Publicado: (2024)

On the Convergence of Adam under Non-uniform Smoothness: Separability from SGDM and Beyond
por: Wang, Bohan, et al.
Publicado: (2024)

Riemannian Optimization for Non-convex Euclidean Distance Geometry with Global Recovery Guarantees
por: Smith, Chandler, et al.
Publicado: (2024)

Adam-SHANG: A Convergent Adam-Type Method for Stochastic Smooth Convex Optimization
por: Yu, Yaxin, et al.
Publicado: (2026)

On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions
por: Hong, Yusu, et al.
Publicado: (2024)

Convergence of Steepest Descent and Adam under Non-Uniform Smoothness
por: Vaswani, Sharan, et al.
Publicado: (2026)

Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction
por: Jiang, Wei, et al.
Publicado: (2024)

Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
por: Yu, Yaxin, et al.
Publicado: (2026)

On the Convergence of Adam-Type Algorithm for Bilevel Optimization under Unbounded Smoothness
por: Gong, Xiaochuan, et al.
Publicado: (2025)

Stochastic Compositional Minimax Optimization with Provable Convergence Guarantees
por: Deng, Yuyang, et al.
Publicado: (2024)

Subspace Optimization for Large Language Models with Convergence Guarantees
por: He, Yutong, et al.
Publicado: (2024)

Online Non-convex Optimization with Long-term Non-convex Constraints
por: Pan, Shijie, et al.
Publicado: (2023)

Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noise
por: Upadhyay, Antesh, et al.
Publicado: (2026)

Adam Converges Without Any Modification On Update Rules
por: Zhang, Yushun, et al.
Publicado: (2026)

Quantization through Piecewise-Affine Regularization: Optimization and Statistical Guarantees
por: Ma, Jianhao, et al.
Publicado: (2025)

Convergence rates for the Adam optimizer
por: Dereich, Steffen, et al.
Publicado: (2024)

Provable Adaptivity of Adam under Non-uniform Smoothness
por: Wang, Bohan, et al.
Publicado: (2022)

Convergence and Complexity Guarantee for Inexact First-order Riemannian Optimization Algorithms
por: Li, Yuchen, et al.
Publicado: (2024)

Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping
por: Liu, Zijian, et al.
Publicado: (2024)

Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise
por: Ahn, Kwangjun, et al.
Publicado: (2024)

A Theoretical and Empirical Study on the Convergence of Adam with an "Exact" Constant Step Size in Non-Convex Settings
por: Mazumder, Alokendu, et al.
Publicado: (2023)

HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization
por: Huang, Feihu, et al.
Publicado: (2026)

Non-convex Stochastic Composite Optimization with Polyak Momentum
por: Gao, Yuan, et al.
Publicado: (2024)

Extended convexity and smoothness and their applications in deep learning
por: Qi, Binchuan, et al.
Publicado: (2024)

Memory-Reduced Meta-Learning with Guaranteed Convergence
por: Yang, Honglin, et al.
Publicado: (2024)

MAP Estimation with Denoisers: Convergence Rates and Guarantees
por: Pesme, Scott, et al.
Publicado: (2025)

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD
por: Jin, Ruinan, et al.
Publicado: (2024)

Retraction-Free Decentralized Non-convex Optimization with Orthogonal Constraints
por: Sun, Youbang, et al.
Publicado: (2024)

Divergence Results and Convergence of a Variance Reduced Version of ADAM
por: Wang, Ruiqi, et al.
Publicado: (2022)

Learning Over-Relaxation Policies for ADMM with Convergence Guarantees
por: Lin, Junan, et al.
Publicado: (2026)

A Stochastic Quasi-Newton Method for Non-convex Optimization with Non-uniform Smoothness
por: Sun, Zhenyu, et al.
Publicado: (2024)

Learning to Optimize for Mixed-Integer Non-linear Programming with Feasibility Guarantees
por: Tang, Bo, et al.
Publicado: (2024)