Copertina

Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Islamov, Rustem, Ajroldi, Niccolo, Orvieto, Antonio, Lucchi, Aurelien
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Optimization and Control
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Loss Landscape Characterization of Neural Networks without Over-Parametrization
di: Islamov, Rustem, et al.
Pubblicazione: (2024)

Why Do We Need Warm-up? A Theoretical Perspective
di: Alimisis, Foivos, et al.
Pubblicazione: (2025)

Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy
di: Islamov, Rustem, et al.
Pubblicazione: (2025)

On the Role of Batch Size in Stochastic Conditional Gradient Methods
di: Islamov, Rustem, et al.
Pubblicazione: (2026)

Byzantine-Robust and Differentially Private Federated Optimization under Weaker Assumptions
di: Islamov, Rustem, et al.
Pubblicazione: (2026)

Safe-EF: Error Feedback for Nonsmooth Constrained Optimization
di: Islamov, Rustem, et al.
Pubblicazione: (2025)

SDEs for Minimax Optimization
di: Compagnoni, Enea Monzio, et al.
Pubblicazione: (2024)

Non-Euclidean Gradient Descent Operates at the Edge of Stability
di: Islamov, Rustem, et al.
Pubblicazione: (2026)

Towards Faster Decentralized Stochastic Optimization with Communication Compression
di: Islamov, Rustem, et al.
Pubblicazione: (2024)

Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
di: Srećković, Teodora, et al.
Pubblicazione: (2025)

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes
di: Orvieto, Antonio, et al.
Pubblicazione: (2024)

Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?
di: Meng, Si Yi, et al.
Pubblicazione: (2025)

Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes
di: Meng, Si Yi, et al.
Pubblicazione: (2024)

Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance
di: Oikonomou, Dimitris, et al.
Pubblicazione: (2024)

Cubic regularized subspace Newton for non-convex optimization
di: Zhao, Jim, et al.
Pubblicazione: (2024)

Recurrent neural networks: vanishing and exploding gradients are not the end of the story
di: Zucchet, Nicolas, et al.
Pubblicazione: (2024)

On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach
di: Compagnoni, Enea Monzio, et al.
Pubblicazione: (2025)

Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise
di: Compagnoni, Enea Monzio, et al.
Pubblicazione: (2024)

When, Where and Why to Average Weights?
di: Ajroldi, Niccolò, et al.
Pubblicazione: (2025)

Stochastic Difference-of-Convex Optimization with Momentum
di: Chayti, El Mahdi, et al.
Pubblicazione: (2025)

Stability and Generalization for Stochastic Recursive Momentum-based Algorithms for (Strongly-)Convex One to $K$-Level Stochastic Optimizations
di: Pan, Xiaokang, et al.
Pubblicazione: (2024)

Shuffling Momentum Gradient Algorithm for Convex Optimization
di: Tran, Trang H., et al.
Pubblicazione: (2024)

Non-convex Stochastic Composite Optimization with Polyak Momentum
di: Gao, Yuan, et al.
Pubblicazione: (2024)

Adaptive Optimization via Momentum on Variance-Normalized Gradients
di: Patitucci, Francisco, et al.
Pubblicazione: (2026)

On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization
di: Sahu, Sharan, et al.
Pubblicazione: (2026)

Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization
di: Liu, Wei, et al.
Pubblicazione: (2025)

Stochastic Compositional Optimization via Hybrid Momentum Frank--Wolfe
di: Chayti, El Mahdi
Pubblicazione: (2026)

Random Scaling and Momentum for Non-smooth Non-convex Optimization
di: Zhang, Qinzi, et al.
Pubblicazione: (2024)

Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration
di: Khaled, Ahmed, et al.
Pubblicazione: (2025)

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models
di: Xie, Xingyu, et al.
Pubblicazione: (2022)

Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum
di: Cai, Haoyuan, et al.
Pubblicazione: (2024)

Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGD
di: Emmanouilidis, Konstantinos, et al.
Pubblicazione: (2026)

Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order
di: Petrov, Egor, et al.
Pubblicazione: (2025)

RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization
di: Chayti, El Mahdi
Pubblicazione: (2026)

Regret-Optimal Federated Transfer Learning for Kernel Regression with Applications in American Option Pricing
di: Yang, Xuwei, et al.
Pubblicazione: (2023)

Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs
di: Compagnoni, Enea Monzio, et al.
Pubblicazione: (2025)

$μ^2$-SGD: Stable Stochastic Optimization via a Double Momentum Mechanism
di: Dahan, Tehila, et al.
Pubblicazione: (2023)

Momentum Further Constrains Sharpness at the Edge of Stochastic Stability
di: Andreyev, Arseniy, et al.
Pubblicazione: (2026)

Algorithmic Stability of Stochastic Gradient Descent with Momentum under Heavy-Tailed Noise
di: Dang, Thanh, et al.
Pubblicazione: (2025)

Constrained Online Convex Optimization with Polyak Feasibility Steps
di: Hutchinson, Spencer, et al.
Pubblicazione: (2025)