Copertina

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Abreu, Natalie, Vyas, Nikhil, Kakade, Sham, Morwani, Depen
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
di: Morwani, Depen, et al.
Pubblicazione: (2025)

Deconstructing What Makes a Good Optimizer for Language Models
di: Zhao, Rosie, et al.
Pubblicazione: (2024)

Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
di: Liu, Bingbin, et al.
Pubblicazione: (2025)

SOAP: Improving and Stabilizing Shampoo using Adam
di: Vyas, Nikhil, et al.
Pubblicazione: (2024)

How Does Critical Batch Size Scale in Pre-training?
di: Zhang, Hanlin, et al.
Pubblicazione: (2024)

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
di: Meterez, Alexandru, et al.
Pubblicazione: (2026)

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
di: Vyas, Nikhil, et al.
Pubblicazione: (2023)

A New Perspective on Shampoo's Preconditioner
di: Morwani, Depen, et al.
Pubblicazione: (2024)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

Loss-to-Loss Prediction: Scaling Laws for All Datasets
di: Brandfonbrener, David, et al.
Pubblicazione: (2024)

LOTION: Smoothing the Optimization Landscape for Quantized Training
di: Kwun, Mujin, et al.
Pubblicazione: (2025)

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
di: Oncescu, Costin-Andrei, et al.
Pubblicazione: (2026)

A Simplified Analysis of SGD for Linear Regression with Weight Averaging
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

Feature emergence via margin maximization: case studies in algebraic tasks
di: Morwani, Depen, et al.
Pubblicazione: (2023)

Learning Hidden Markov Models Using Conditional Samples
di: Kakade, Sham M., et al.
Pubblicazione: (2023)

Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond
di: Oncescu, Costin-Andrei, et al.
Pubblicazione: (2024)

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
di: Jin, Jikai, et al.
Pubblicazione: (2025)

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
di: Zhang, Hanlin, et al.
Pubblicazione: (2026)

Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization
di: Buffelli, Davide, et al.
Pubblicazione: (2024)

Soup to go: mitigating forgetting during continual learning with model averaging
di: Kleiman, Anat, et al.
Pubblicazione: (2025)

Repeat After Me: Transformers are Better than State Space Models at Copying
di: Jelassi, Samy, et al.
Pubblicazione: (2024)

Scaling Laws for Imitation Learning in Single-Agent Games
di: Tuyls, Jens, et al.
Pubblicazione: (2023)

Comparing BFGS and OGR for Second-Order Optimization
di: Przybysz, Adrian, et al.
Pubblicazione: (2025)

Taming Preconditioner Drift: Unlocking the Potential of Second-Order Optimizers for Federated Learning on Non-IID Data
di: Liu, Junkang, et al.
Pubblicazione: (2026)

A Study on the Calibration of In-context Learning
di: Zhang, Hanlin, et al.
Pubblicazione: (2023)

The Role of Sparsity for Length Generalization in Transformers
di: Golowich, Noah, et al.
Pubblicazione: (2025)

CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
di: Brandfonbrener, David, et al.
Pubblicazione: (2024)

A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networks
di: Cayci, Semih
Pubblicazione: (2024)

Second-Order Convergence in Private Stochastic Non-Convex Optimization
di: Tao, Youming, et al.
Pubblicazione: (2025)

Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
di: Qi, Zhenting, et al.
Pubblicazione: (2024)

Scaling Laws in Linear Regression: Compute, Parameters, and Data
di: Lin, Licong, et al.
Pubblicazione: (2024)

Simultaneous Training of First- and Second-Order Optimizers in Population-Based Reinforcement Learning
di: Pfeiffer, Felix, et al.
Pubblicazione: (2024)

Transcendence: Generative Models Can Outperform The Experts That Train Them
di: Zhang, Edwin, et al.
Pubblicazione: (2024)

From Alignment to Prediction: A Study of Self-Supervised Learning and Predictive Representation Learning
di: Dutta, Mintu, et al.
Pubblicazione: (2026)

Deep Learning with Tabular Data: A Self-supervised Approach
di: Vyas, Tirth Kiranbhai
Pubblicazione: (2024)

Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment
di: Pustejovsky, James, et al.
Pubblicazione: (2026)

A Variance-Reduced Cubic-Regularized Newton for Policy Optimization
di: Sun, Cheng, et al.
Pubblicazione: (2025)

A Second-Order Perspective on Model Compositionality and Incremental Learning
di: Porrello, Angelo, et al.
Pubblicazione: (2024)

The Newton-Muon Optimizer
di: Du, Zhehang, et al.
Pubblicazione: (2026)

Explaining Predictive Uncertainty by Exposing Second-Order Effects
di: Bley, Florian, et al.
Pubblicazione: (2024)