Copertina

Beyond First-Order: Training LLMs with Stochastic Conjugate Subgradients and AdamW

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhang, Di, Zhang, Yihang
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Optimizer-Induced Mode Connectivity: From AdamW to Muon
di: Zhang, Fangzhao, et al.
Pubblicazione: (2026)

APOLLO: SGD-like Memory, AdamW-level Performance
di: Zhu, Hanqing, et al.
Pubblicazione: (2024)

First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation
di: Khanh, Truong Xuan, et al.
Pubblicazione: (2026)

How to set AdamW's weight decay as you scale model and dataset size
di: Wang, Xi, et al.
Pubblicazione: (2024)

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
di: Ranganath, Aditya
Pubblicazione: (2026)

Uniform Scaling Limits in AdamW-Trained Transformers
di: Gibson, William, et al.
Pubblicazione: (2026)

The Stochastic Conjugate Subgradient Algorithm For Kernel Support Vector Machines
di: Zhang, Di, et al.
Pubblicazione: (2024)

HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization
di: Huang, Feihu, et al.
Pubblicazione: (2026)

Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks
di: Tang, Xuan, et al.
Pubblicazione: (2025)

Anon: Extrapolating Adaptivity Beyond SGD and Adam
di: Zhang, Yiheng, et al.
Pubblicazione: (2026)

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization
di: Xiao, Nachuan, et al.
Pubblicazione: (2023)

In-Run Data Shapley for Adam Optimizer
di: Ding, Meng, et al.
Pubblicazione: (2026)

Order-Preserving GFlowNets
di: Chen, Yihang, et al.
Pubblicazione: (2023)

Second-Order Convergence in Private Stochastic Non-Convex Optimization
di: Tao, Youming, et al.
Pubblicazione: (2025)

SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam
di: Peng, Hanyang, et al.
Pubblicazione: (2025)

Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization
di: Xie, Shuo, et al.
Pubblicazione: (2024)

AB-Cache: Training-Free Acceleration of Diffusion Models via Adams-Bashforth Cached Feature Reuse
di: Yu, Zichao, et al.
Pubblicazione: (2025)

Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning
di: Pfeiffer, Felix, et al.
Pubblicazione: (2024)

Conda: Column-Normalized Adam for Training Large Language Models Faster
di: Wang, Junjie, et al.
Pubblicazione: (2025)

BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
di: Zhao, Zhixiong, et al.
Pubblicazione: (2026)

DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models
di: Liu, Jin, et al.
Pubblicazione: (2026)

The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizers
di: Liu, Qi, et al.
Pubblicazione: (2024)

DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning
di: Chooi, Jay, et al.
Pubblicazione: (2025)

Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs
di: Mukherjee, Sagnik, et al.
Pubblicazione: (2026)

FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models
di: Liu, Junkang, et al.
Pubblicazione: (2025)

On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks
di: Gao, Yihang, et al.
Pubblicazione: (2024)

Dimer-Enhanced Optimization: A First-Order Approach to Escaping Saddle Points in Neural Network Training
di: Hu, Yue, et al.
Pubblicazione: (2025)

Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
di: Wen, Ziqing, et al.
Pubblicazione: (2026)

FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
di: Dang, Sizhe, et al.
Pubblicazione: (2025)

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
di: Cai, Zhijie, et al.
Pubblicazione: (2026)

Why Transformers Need Adam: A Hessian Perspective
di: Zhang, Yushun, et al.
Pubblicazione: (2024)

AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
di: Zhang, Huishuai, et al.
Pubblicazione: (2025)

Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training
di: Ye, Chenlu, et al.
Pubblicazione: (2025)

Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks
di: Xiao, Mingqing, et al.
Pubblicazione: (2024)

A Universal Banach--Bregman Framework for Stochastic Iterations: Unifying Stochastic Mirror Descent, Learning and LLM Training
di: Zhang, Johnny R., et al.
Pubblicazione: (2025)

Understanding Adam Requires Better Rotation Dependent Assumptions
di: Zhang, Tianyue H., et al.
Pubblicazione: (2024)

Beyond Training: Enabling Self-Evolution of Agents with MOBIMEM
di: Liu, Zibin, et al.
Pubblicazione: (2025)

D$^2$Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs
di: Yan, Xianglong, et al.
Pubblicazione: (2026)

Unraveling Text Generation in LLMs: A Stochastic Differential Equation Approach
di: Zhang, Yukun
Pubblicazione: (2024)

Benign Overfitting in Adversarial Training for Vision Transformers
di: Zhang, Jiaming, et al.
Pubblicazione: (2026)