Copertina

Why Transformers Need Adam: A Hessian Perspective

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhang, Yushun, Chen, Congliang, Ding, Tian, Li, Ziniu, Sun, Ruoyu, Luo, Zhi-Quan
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Adam-mini: Use Fewer Learning Rates To Gain More
di: Zhang, Yushun, et al.
Pubblicazione: (2024)

Preserving Diversity in Supervised Fine-Tuning of Large Language Models
di: Li, Ziniu, et al.
Pubblicazione: (2024)

Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
di: Li, Ziniu, et al.
Pubblicazione: (2025)

Adam Converges Without Any Modification On Update Rules
di: Zhang, Yushun, et al.
Pubblicazione: (2026)

MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
di: Chen, Yupeng, et al.
Pubblicazione: (2024)

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
di: Li, Ziniu, et al.
Pubblicazione: (2023)

Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborative
di: Zhang, Tuo, et al.
Pubblicazione: (2025)

Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective
di: Wang, Senmiao, et al.
Pubblicazione: (2025)

Towards Quantifying the Hessian Structure of Neural Networks
di: Dong, Zhaorui, et al.
Pubblicazione: (2025)

Provable Adaptivity of Adam under Non-uniform Smoothness
di: Wang, Bohan, et al.
Pubblicazione: (2022)

Graph Neural Networks Are More Than Filters: Revisiting and Benchmarking from A Spectral Perspective
di: Dong, Yushun, et al.
Pubblicazione: (2024)

In-Run Data Shapley for Adam Optimizer
di: Ding, Meng, et al.
Pubblicazione: (2026)

ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood
di: Wang, Ruoyu, et al.
Pubblicazione: (2024)

A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectives
di: Zhao, Kaixiang, et al.
Pubblicazione: (2025)

CoRT: Code-integrated Reasoning within Thinking
di: Li, Chengpeng, et al.
Pubblicazione: (2025)

No More Adam: Learning Rate Scaling at Initialization is All You Need
di: Xu, Minghao, et al.
Pubblicazione: (2024)

Independence Constrained Disentangled Representation Learning from Epistemological Perspective
di: Wang, Ruoyu, et al.
Pubblicazione: (2024)

Self-Evolving Critique Abilities in Large Language Models
di: Tang, Zhengyang, et al.
Pubblicazione: (2025)

RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques
di: Tang, Zhengyang, et al.
Pubblicazione: (2025)

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds
di: Liang, Hao, et al.
Pubblicazione: (2022)

Why Adam Can Beat SGD: Second-Moment Normalization Yields Sharper Tails
di: Jin, Ruinan, et al.
Pubblicazione: (2026)

ST-FiT: Inductive Spatial-Temporal Forecasting with Limited Training Data
di: Lei, Zhenyu, et al.
Pubblicazione: (2024)

Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs
di: Mukherjee, Sagnik, et al.
Pubblicazione: (2026)

Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
di: Wen, Ziqing, et al.
Pubblicazione: (2026)

$XX^{t}$ Can Be Faster
di: Rybin, Dmitry, et al.
Pubblicazione: (2025)

Why Adam Works Better with $β_1 = β_2$: The Missing Gradient Scale Invariance Principle
di: Fernández-Hernández, Alberto, et al.
Pubblicazione: (2026)

HeRo-Q: A General Framework for Stable Low Bit Quantization via Hessian Conditioning
di: Zhang, Jinhao Zhang Yunquan, et al.
Pubblicazione: (2026)

Hessian-Free Online Certified Unlearning
di: Qiao, Xinbao, et al.
Pubblicazione: (2024)

Position: Why a Dynamical Systems Perspective is Needed to Advance Time Series Modeling
di: Durstewitz, Daniel, et al.
Pubblicazione: (2026)

Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
di: Ke, Yekun, et al.
Pubblicazione: (2024)

Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent
di: Li, Yingru, et al.
Pubblicazione: (2024)

AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
di: Zhang, Huishuai, et al.
Pubblicazione: (2025)

A Note on Hybrid Online Reinforcement and Imitation Learning for LLMs: Formulations and Algorithms
di: Li, Yingru, et al.
Pubblicazione: (2025)

COMPOL: A Unified Neural Operator Framework for Scalable Multi-Physics Simulations
di: Sun, Yifei, et al.
Pubblicazione: (2025)

PENGUIN: Enhancing Transformer with Periodic-Nested Group Attention for Long-term Time Series Forecasting
di: Sun, Tian, et al.
Pubblicazione: (2025)

What Matters in Transformers? Not All Attention is Needed
di: He, Shwai, et al.
Pubblicazione: (2024)

Dataset Distillation for Offline Reinforcement Learning
di: Light, Jonathan, et al.
Pubblicazione: (2024)

Neighbourhood Transformer: Switchable Attention for Monophily-Aware Graph Learning
di: Luo, Yi, et al.
Pubblicazione: (2026)

Why Do Time Series Models Need Long Context Windows?
di: Butera, Luca, et al.
Pubblicazione: (2026)

WarpAdam: A new Adam optimizer based on Meta-Learning approach
di: Pan, Chengxi, et al.
Pubblicazione: (2024)