Copertina

Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Liu, Bingbin, Bansal, Rachit, Morwani, Depen, Vyas, Nikhil, Alvarez-Melis, David, Kakade, Sham M.
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
di: Abreu, Natalie, et al.
Pubblicazione: (2025)

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
di: Morwani, Depen, et al.
Pubblicazione: (2025)

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
di: Vyas, Nikhil, et al.
Pubblicazione: (2023)

Deconstructing What Makes a Good Optimizer for Language Models
di: Zhao, Rosie, et al.
Pubblicazione: (2024)

SOAP: Improving and Stabilizing Shampoo using Adam
di: Vyas, Nikhil, et al.
Pubblicazione: (2024)

A New Perspective on Shampoo's Preconditioner
di: Morwani, Depen, et al.
Pubblicazione: (2024)

A Simplified Analysis of SGD for Linear Regression with Weight Averaging
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

How Does Critical Batch Size Scale in Pre-training?
di: Zhang, Hanlin, et al.
Pubblicazione: (2024)

LOTION: Smoothing the Optimization Landscape for Quantized Training
di: Kwun, Mujin, et al.
Pubblicazione: (2025)

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
di: Meterez, Alexandru, et al.
Pubblicazione: (2026)

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
di: Oncescu, Costin-Andrei, et al.
Pubblicazione: (2026)

Feature emergence via margin maximization: case studies in algebraic tasks
di: Morwani, Depen, et al.
Pubblicazione: (2023)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

Loss-to-Loss Prediction: Scaling Laws for All Datasets
di: Brandfonbrener, David, et al.
Pubblicazione: (2024)

In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
di: Panigrahi, Abhishek, et al.
Pubblicazione: (2025)

Mixture of Parrots: Experts improve memorization more than reasoning
di: Jelassi, Samy, et al.
Pubblicazione: (2024)

Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
di: Kim, Jaeyeon, et al.
Pubblicazione: (2026)

Random Scaling of Emergent Capabilities
di: Zhao, Rosie, et al.
Pubblicazione: (2025)

Characterization and Mitigation of Training Instabilities in Microscaling Formats
di: Su, Huangyuan, et al.
Pubblicazione: (2025)

GQ-VAE: A gated quantized VAE for learning variable length tokens
di: Datta, Theo, et al.
Pubblicazione: (2025)

Universal Length Generalization with Turing Programs
di: Hou, Kaiying, et al.
Pubblicazione: (2024)

The Rich and the Simple: On the Implicit Bias of Adam and SGD
di: Vasudeva, Bhavya, et al.
Pubblicazione: (2025)

Anon: Extrapolating Adaptivity Beyond SGD and Adam
di: Zhang, Yiheng, et al.
Pubblicazione: (2026)

APOLLO: SGD-like Memory, AdamW-level Performance
di: Zhu, Hanqing, et al.
Pubblicazione: (2024)

Repeat After Me: Transformers are Better than State Space Models at Copying
di: Jelassi, Samy, et al.
Pubblicazione: (2024)

Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
di: Huang, Jing, et al.
Pubblicazione: (2026)

Gauss-Newton Unlearning for the LLM Era
di: McKinney, Lev, et al.
Pubblicazione: (2026)

Measures of Information Reflect Memorization Patterns
di: Bansal, Rachit, et al.
Pubblicazione: (2022)

Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
di: Sahu, Sharan, et al.
Pubblicazione: (2026)

Error whitening: Why Gauss-Newton outperforms Newton
di: McKay, Maricela Best, et al.
Pubblicazione: (2026)

Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
di: Kim, Jaeyeon, et al.
Pubblicazione: (2025)

Guided Speculative Inference for Efficient Test-Time Alignment of LLMs
di: Geuter, Jonathan, et al.
Pubblicazione: (2025)

Q-Probe: A Lightweight Approach to Reward Maximization for Language Models
di: Li, Kenneth, et al.
Pubblicazione: (2024)

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
di: Zhang, Hanlin, et al.
Pubblicazione: (2026)

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
di: Jin, Jikai, et al.
Pubblicazione: (2025)

Learning Hidden Markov Models Using Conditional Samples
di: Kakade, Sham M., et al.
Pubblicazione: (2023)

Matching the Statistical Query Lower Bound for $k$-Sparse Parity Problems with Sign Stochastic Gradient Descent
di: Kou, Yiwen, et al.
Pubblicazione: (2024)

Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond
di: Oncescu, Costin-Andrei, et al.
Pubblicazione: (2024)

Incremental Gauss--Newton Methods with Superlinear Convergence Rates
di: Zhou, Zhiling, et al.
Pubblicazione: (2024)

Fast Gauss-Newton for Multiclass Cross-Entropy
di: Korbit, Mikalai, et al.
Pubblicazione: (2026)