Imagen de Portada

Cautious Weight Decay

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Chen, Lizhang, Li, Jonathan, Liang, Kaizhao, Su, Baiyu, Xie, Cong, Pierse, Nuo Wang, Liang, Chen, Lao, Ni, Liu, Qiang
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning Optimization and Control
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Lion Secretly Solves Constrained Optimization: As Lyapunov Predicts
por: Chen, Lizhang, et al.
Publicado: (2023)

$ϕ$-Balancing for Mixture-of-Experts Training
por: Chen, Lizhang, et al.
Publicado: (2026)

Muon Optimizes Under Spectral Norm Constraints
por: Chen, Lizhang, et al.
Publicado: (2025)

Cautious Optimizers: Improving Training with One Line of Code
por: Liang, Kaizhao, et al.
Publicado: (2024)

Communication Efficient Distributed Training with Distributed Lion
por: Liu, Bo, et al.
Publicado: (2024)

CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization
por: Huang, Feihu, et al.
Publicado: (2026)

Does Weight Decay Enhance Training Stability?
por: Saether, Marius, et al.
Publicado: (2026)

Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse
por: Jacot, Arthur, et al.
Publicado: (2024)

Training-Free Looped Transformers
por: Chen, Lizhang, et al.
Publicado: (2026)

A Weighted Gradient Tracking Privacy-Preserving Method for Distributed Optimization
por: Xie, Furan, et al.
Publicado: (2025)

Modified Meta-Thompson Sampling for Linear Bandits and Its Bayes Regret Analysis
por: Li, Hao, et al.
Publicado: (2024)

Memory-Efficient LLM Training with Online Subspace Descent
por: Liang, Kaizhao, et al.
Publicado: (2024)

Proximal Oracles for Optimization and Sampling
por: Liang, Jiaming, et al.
Publicado: (2024)

Solving Inverse Problems with Deep Linear Neural Networks: Global Convergence Guarantees for Gradient Descent with Weight Decay
por: Laus, Hannah, et al.
Publicado: (2025)

Propagation of Chaos in Contextual Flow Maps
por: Chen, Shi, et al.
Publicado: (2026)

On the Benefits of Weight Normalization for Overparameterized Matrix Sensing
por: Wei, Yudong, et al.
Publicado: (2025)

The Local Landscape of Phase Retrieval Under Limited Samples
por: Liu, Kaizhao, et al.
Publicado: (2023)

Faster Rates for No-Regret Learning in General Games via Cautious Optimism
por: Soleymani, Ashkan, et al.
Publicado: (2025)

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization
por: Zhang, Haihan, et al.
Publicado: (2024)

Cautious Optimism: A Meta-Algorithm for Near-Constant Regret in General Games
por: Soleymani, Ashkan, et al.
Publicado: (2025)

Accelerating Single-Pass SGD for Generalized Linear Prediction
por: Chen, Qian, et al.
Publicado: (2026)

Gradient descent in matrix factorization: Understanding large initialization
por: Chen, Hengchao, et al.
Publicado: (2023)

Variance Reduction and Low Sample Complexity in Stochastic Optimization via Proximal Point Method
por: Liang, Jiaming
Publicado: (2024)

A Unified Analysis for Finite Weight Averaging
por: Wang, Peng, et al.
Publicado: (2024)

Diffusion Model for Data-Driven Black-Box Optimization
por: Li, Zihao, et al.
Publicado: (2024)

Convergence rates of stochastic gradient method with independent sequences of step-size and momentum weight
por: Hwang, Wen-Liang
Publicado: (2024)

Decoupled Weight Decay for Any $p$ Norm
por: Outmezguine, Nadav Joseph, et al.
Publicado: (2024)

Offline Policy Learning with Weight Clipping and Heaviside Composite Optimization
por: Liu, Jingren, et al.
Publicado: (2026)

Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization
por: Armacki, Aleksandar, et al.
Publicado: (2026)

Trade-off in Estimating the Number of Byzantine Clients in Federated Learning
por: Chen, Ziyi, et al.
Publicado: (2025)

On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimization
por: Liang, Ling, et al.
Publicado: (2024)

A Provably Convergent and Practical Algorithm for Gromov--Wasserstein Optimal Transport
por: Liang, Ling, et al.
Publicado: (2026)

An inexact Bregman proximal point method and its acceleration version for unbalanced optimal transport
por: Chen, Xiang, et al.
Publicado: (2024)

Inertial Quadratic Majorization Minimization with Application to Kernel Regularized Learning
por: Heng, Qiang, et al.
Publicado: (2025)

Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs
por: Liang, Yifei, et al.
Publicado: (2026)

A Hessian-Aware Stochastic Differential Equation for Modelling SGD
por: Li, Xiang, et al.
Publicado: (2024)

From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees
por: Xie, Shengping, et al.
Publicado: (2025)

Bias and Extrapolation in Markovian Linear Stochastic Approximation with Constant Stepsizes
por: Huo, Dongyan, et al.
Publicado: (2022)

Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
por: Harada, Hinata, et al.
Publicado: (2024)

Nonconvex Optimization Framework for Group-Sparse Feedback Linear-Quadratic Optimal Control: Non-Penalty Approach
por: Feng, Lechen, et al.
Publicado: (2025)