Cover Image

AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Huishuai, Wang, Bohan, Chen, Luoxin
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Learning to Generate Formally Verifiable Step-by-Step Logic Reasoning via Structured Formal Intermediaries
by: Chen, Luoxin, et al.
Published: (2026)

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
by: Li, Mengqi, et al.
Published: (2025)

SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training
by: Huang, Tianjin, et al.
Published: (2025)

Your Agent Can Defend Itself against Backdoor Attacks
by: Changjiang, Li, et al.
Published: (2025)

Data Can Speak for Itself: Quality-guided Utilization of Wireless Synthetic Data
by: Gong, Chen, et al.
Published: (2025)

Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails
by: Jin, Ruinan, et al.
Published: (2026)

OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases
by: Wang, Zixiao, et al.
Published: (2026)

Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training
by: Barakat, Anas, et al.
Published: (2026)

Conda: Column-Normalized Adam for Training Large Language Models Faster
by: Wang, Junjie, et al.
Published: (2025)

Tracing the Representation Geometry of Language Models from Pretraining to Post-training
by: Li, Melody Zixuan, et al.
Published: (2025)

KL-regularization Itself is Differentially Private in Bandits and RLHF
by: Zhang, Yizhou, et al.
Published: (2025)

When AI Eats Itself: On the Caveats of AI Autophagy
by: Xing, Xiaodan, et al.
Published: (2024)

Bridging Distribution Gaps in Time Series Foundation Model Pretraining with Prototype-Guided Normalization
by: Gong, Peiliang, et al.
Published: (2025)

Solving Formal Math Problems by Decomposition and Iterative Reflection
by: Zhou, Yichi, et al.
Published: (2025)

MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
by: Chen, Yupeng, et al.
Published: (2024)

Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
by: Liu, Qihao, et al.
Published: (2025)

PostTrainBench: Can LLM Agents Automate LLM Post-Training?
by: Rank, Ben, et al.
Published: (2026)

Let the Code LLM Edit Itself When You Edit the Code
by: He, Zhenyu, et al.
Published: (2024)

Post-training an LLM for RAG? Train on Self-Generated Demonstrations
by: Finlayson, Matthew, et al.
Published: (2025)

Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess
by: Hwang, Dongyoon, et al.
Published: (2025)

LoReUn: Data Itself Implicitly Provides Cues to Improve Machine Unlearning
by: Li, Xiang, et al.
Published: (2025)

In-Run Data Shapley for Adam Optimizer
by: Ding, Meng, et al.
Published: (2026)

ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
by: You, Haoran, et al.
Published: (2024)

Why Transformers Need Adam: A Hessian Perspective
by: Zhang, Yushun, et al.
Published: (2024)

CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data
by: Xie, Shifeng, et al.
Published: (2025)

The Impact of Post-training on Data Contamination
by: Kocyigit, Muhammed Yusuf, et al.
Published: (2026)

Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
by: Akter, Syeda Nahida, et al.
Published: (2025)

Can GRPO Help LLMs Transcend Their Pretraining Origin?
by: Ni, Kangqi, et al.
Published: (2025)

LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimization
by: Zhang, Zishi, et al.
Published: (2026)

AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
by: Cai, Zhijie, et al.
Published: (2026)

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
by: Yan, Peishen, et al.
Published: (2026)

Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
by: Elgaar, Mohamed, et al.
Published: (2026)

Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders
by: Jing, Yi, et al.
Published: (2026)

WarpAdam: A new Adam optimizer based on Meta-Learning approach
by: Pan, Chengxi, et al.
Published: (2024)

Anon: Extrapolating Adaptivity Beyond SGD and Adam
by: Zhang, Yiheng, et al.
Published: (2026)

Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
by: Ranganath, Aditya
Published: (2026)

InterCorpRel-LLM: Enhancing Financial Relational Understanding with Graph-Language Models
by: Sun, Qianyou, et al.
Published: (2025)

Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults
by: Phunyaphibarn, Prin, et al.
Published: (2023)

On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systems
by: Tang, Bohan, et al.
Published: (2025)

BoA: Attention-aware Post-training Quantization without Backpropagation
by: Kim, Junhan, et al.
Published: (2024)