Image de couverture de livre

Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Jin, Ruinan, Liang, Yingbin, Zou, Shaofeng
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Anon: Extrapolating Adaptivity Beyond SGD and Adam
par: Zhang, Yiheng, et autres
Publié: (2026)

Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation
par: Chen, Ziru, et autres
Publié: (2026)

AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
par: Zhang, Huishuai, et autres
Publié: (2025)

APOLLO: SGD-like Memory, AdamW-level Performance
par: Zhu, Hanqing, et autres
Publié: (2024)

Sharper Generalization Bounds for Transformer
par: Li, Yawen, et autres
Publié: (2026)

HIPO: Instruction Hierarchy via Constrained Reinforcement Learning
par: Chen, Keru, et autres
Publié: (2026)

Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs
par: Mukherjee, Sagnik, et autres
Publié: (2026)

A Sharper Picture of Generalization in Transformers
par: Lintilhac, Paul, et autres
Publié: (2026)

A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD
par: Jin, Ruinan, et autres
Publié: (2024)

SWAN: SGD with Normalization and Whitening Enables Stateless LLM Training
par: Ma, Chao, et autres
Publié: (2024)

Why Transformers Need Adam: A Hessian Perspective
par: Zhang, Yushun, et autres
Publié: (2024)

SoftSignSGD(S3): An Enhanced Optimizer for Practical DNN Training and Loss Spikes Minimization Beyond Adam
par: Peng, Hanyang, et autres
Publié: (2025)

Sharper Error Bounds in Late Fusion Multi-view Clustering Using Eigenvalue Proportion
par: Du, Liang, et autres
Publié: (2024)

Muon Outperforms Adam in Tail-End Associative Memory Learning
par: Wang, Shuche, et autres
Publié: (2025)

Conda: Column-Normalized Adam for Training Large Language Models Faster
par: Wang, Junjie, et autres
Publié: (2025)

Model-Free Robust Reinforcement Learning with Sample Complexity Analysis
par: Wang, Yudan, et autres
Publié: (2024)

Near-Optimal Partially Observable Reinforcement Learning with Partial Online State Information
par: Shi, Ming, et autres
Publié: (2023)

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
par: Cai, Zhijie, et autres
Publié: (2026)

Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning
par: Liu, Ziyue, et autres
Publié: (2026)

Can LLMs Effectively Leverage Graph Structural Information through Prompts, and Why?
par: Huang, Jin, et autres
Publié: (2023)

Detector-Evasive LLM Paraphrasing via Constrained Policy Optimization
par: Wang, Mingyi, et autres
Publié: (2026)

Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation
par: Wang, Yudan, et autres
Publié: (2024)

Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks
par: Tang, Xuan, et autres
Publié: (2025)

Representations learnt by SGD and Adaptive learning rules: Conditions that vary sparsity and selectivity in neural networks
par: Park, Jin Hyun
Publié: (2022)

Why Adam Works Better with $β_1 = β_2$: The Missing Gradient Scale Invariance Principle
par: Fernández-Hernández, Alberto, et autres
Publié: (2026)

Federated Learning on Virtual Heterogeneous Data with Local-global Distillation
par: Huang, Chun-Yin, et autres
Publié: (2023)

Learnable Chernoff Baselines for Inference-Time Alignment
par: Madhow, Sunil, et autres
Publié: (2026)

When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds
par: Tao, Hongyi, et autres
Publié: (2026)

Why the Maximum Second Derivative of Activations Matters for Adversarial Robustness
par: Yu, Yunrui, et autres
Publié: (2026)

Bootstrap SGD: Algorithmic Stability and Robustness
par: Christmann, Andreas, et autres
Publié: (2024)

Learning Efficiency Meets Symmetry Breaking
par: Bai, Yingbin, et autres
Publié: (2025)

Beating Transformers using Synthetic Cognition
par: Ibias, Alfredo, et autres
Publié: (2025)

Theory on Mixture-of-Experts in Continual Learning
par: Li, Hongbo, et autres
Publié: (2024)

Why Self-Supervised Encoders Want to Be Normal
par: Domb, Yuval
Publié: (2026)

Accumulative SGD Influence Estimation for Data Attribution
par: Shi, Yunxiao, et autres
Publié: (2025)

Beat Tracking as Object Detection
par: Ahn, Jaehoon, et autres
Publié: (2025)

Agentic Transformers Provably Learn to Search via Reinforcement Learning
par: Yang, Tong, et autres
Publié: (2026)

Sample Complexity Characterization for Linear Contextual MDPs
par: Deng, Junze, et autres
Publié: (2024)

Contrastive and Variational Approaches in Self-Supervised Learning for Complex Data Mining
par: Liang, Yingbin, et autres
Publié: (2025)

Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
par: Kim, Jihwan, et autres
Publié: (2026)