Copertina

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Vyas, Nikhil, Morwani, Depen, Zhao, Rosie, Kaplun, Gal, Kakade, Sham, Barak, Boaz
Natura:	Preprint
Pubblicazione:	2023
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
di: Morwani, Depen, et al.
Pubblicazione: (2025)

Deconstructing What Makes a Good Optimizer for Language Models
di: Zhao, Rosie, et al.
Pubblicazione: (2024)

The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
di: Abreu, Natalie, et al.
Pubblicazione: (2025)

Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
di: Liu, Bingbin, et al.
Pubblicazione: (2025)

SOAP: Improving and Stabilizing Shampoo using Adam
di: Vyas, Nikhil, et al.
Pubblicazione: (2024)

A New Perspective on Shampoo's Preconditioner
di: Morwani, Depen, et al.
Pubblicazione: (2024)

A Simplified Analysis of SGD for Linear Regression with Weight Averaging
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

Feature emergence via margin maximization: case studies in algebraic tasks
di: Morwani, Depen, et al.
Pubblicazione: (2023)

How Does Critical Batch Size Scale in Pre-training?
di: Zhang, Hanlin, et al.
Pubblicazione: (2024)

LOTION: Smoothing the Optimization Landscape for Quantized Training
di: Kwun, Mujin, et al.
Pubblicazione: (2025)

Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averaging
di: Meterez, Alexandru, et al.
Pubblicazione: (2026)

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
di: Oncescu, Costin-Andrei, et al.
Pubblicazione: (2026)

Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
di: Meterez, Alexandru, et al.
Pubblicazione: (2025)

Loss-to-Loss Prediction: Scaling Laws for All Datasets
di: Brandfonbrener, David, et al.
Pubblicazione: (2024)

Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
di: Zhao, Rosie, et al.
Pubblicazione: (2025)

Distinguishing the Knowable from the Unknowable with Language Models
di: Ahdritz, Gustaf, et al.
Pubblicazione: (2024)

Random Scaling of Emergent Capabilities
di: Zhao, Rosie, et al.
Pubblicazione: (2025)

The Rich and the Simple: On the Implicit Bias of Adam and SGD
di: Vasudeva, Bhavya, et al.
Pubblicazione: (2025)

Implicit Bias in Noisy-SGD: With Applications to Differentially Private Training
di: Sander, Tom, et al.
Pubblicazione: (2024)

Characterization and Mitigation of Training Instabilities in Microscaling Formats
di: Su, Huangyuan, et al.
Pubblicazione: (2025)

Risk Comparisons in Linear Regression: Implicit Regularization Dominates Explicit Regularization
di: Wu, Jingfeng, et al.
Pubblicazione: (2025)

Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond
di: Oncescu, Costin-Andrei, et al.
Pubblicazione: (2024)

Mixture of Parrots: Experts improve memorization more than reasoning
di: Jelassi, Samy, et al.
Pubblicazione: (2024)

Koopman Spectrum Nonlinear Regulators and Efficient Online Learning
di: Ohnishi, Motoya, et al.
Pubblicazione: (2021)

The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networks
di: Gronich, Eitan, et al.
Pubblicazione: (2026)

Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models
di: Jelassi, Samy, et al.
Pubblicazione: (2026)

Learning Hidden Markov Models Using Conditional Samples
di: Kakade, Sham M., et al.
Pubblicazione: (2023)

GQ-VAE: A gated quantized VAE for learning variable length tokens
di: Datta, Theo, et al.
Pubblicazione: (2025)

Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning
di: Jin, Jikai, et al.
Pubblicazione: (2025)

Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks
di: Tsilivis, Nikolaos, et al.
Pubblicazione: (2024)

Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions
di: Kim, Jaeyeon, et al.
Pubblicazione: (2025)

Universal Length Generalization with Turing Programs
di: Hou, Kaiying, et al.
Pubblicazione: (2024)

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
di: Zhang, Hanlin, et al.
Pubblicazione: (2026)

Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise
di: Bae, Youngkyoung, et al.
Pubblicazione: (2024)

Matching the Statistical Query Lower Bound for $k$-Sparse Parity Problems with Sign Stochastic Gradient Descent
di: Kou, Yiwen, et al.
Pubblicazione: (2024)

Workspace Optimization: How to Train Your Agent
di: Sarafian, Elad, et al.
Pubblicazione: (2026)

Scaling Laws for Imitation Learning in Single-Agent Games
di: Tuyls, Jens, et al.
Pubblicazione: (2023)

Learning an Inventory Control Policy with General Inventory Arrival Dynamics
di: Andaz, Sohrab, et al.
Pubblicazione: (2023)

Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
di: Kim, Jaeyeon, et al.
Pubblicazione: (2026)

How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD
di: Beneventano, Pierfrancesco, et al.
Pubblicazione: (2024)