Imagen de Portada

Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Compagnoni, Enea Monzio, Liu, Tianlin, Islamov, Rustem, Proske, Frank Norbert, Orvieto, Antonio, Lucchi, Aurelien
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs
por: Compagnoni, Enea Monzio, et al.
Publicado: (2025)

On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach
por: Compagnoni, Enea Monzio, et al.
Publicado: (2025)

SDEs for Minimax Optimization
por: Compagnoni, Enea Monzio, et al.
Publicado: (2024)

Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspective
por: Compagnoni, Enea Monzio, et al.
Publicado: (2026)

Loss Landscape Characterization of Neural Networks without Over-Parametrization
por: Islamov, Rustem, et al.
Publicado: (2024)

Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size
por: Islamov, Rustem, et al.
Publicado: (2025)

Why Do We Need Warm-up? A Theoretical Perspective
por: Alimisis, Foivos, et al.
Publicado: (2025)

On the Role of Batch Size in Stochastic Conditional Gradient Methods
por: Islamov, Rustem, et al.
Publicado: (2026)

Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy
por: Islamov, Rustem, et al.
Publicado: (2025)

Byzantine-Robust and Differentially Private Federated Optimization under Weaker Assumptions
por: Islamov, Rustem, et al.
Publicado: (2026)

Safe-EF: Error Feedback for Nonsmooth Constrained Optimization
por: Islamov, Rustem, et al.
Publicado: (2025)

Explaining Grokking in Transformers through the Lens of Inductive Bias
por: Singh, Jaisidh, et al.
Publicado: (2026)

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes
por: Orvieto, Antonio, et al.
Publicado: (2024)

Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks
por: Zhao, Jim, et al.
Publicado: (2024)

A Theoretical Analysis of the Learning Dynamics under Class Imbalance
por: Francazi, Emanuele, et al.
Publicado: (2022)

Towards Faster Decentralized Stochastic Optimization with Communication Compression
por: Islamov, Rustem, et al.
Publicado: (2024)

Non-Euclidean Gradient Descent Operates at the Edge of Stability
por: Islamov, Rustem, et al.
Publicado: (2026)

Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture
por: Movahedi, Sajad, et al.
Publicado: (2024)

Adam Simplified: Bias Correction Debunked
por: Laing, Sam, et al.
Publicado: (2025)

Revisiting associative recall in modern recurrent models
por: Okpekpe, Destiny, et al.
Publicado: (2025)

Cubic regularized subspace Newton for non-convex optimization
por: Zhao, Jim, et al.
Publicado: (2024)

In Search of Adam's Secret Sauce
por: Orvieto, Antonio, et al.
Publicado: (2025)

Theoretical Foundations of Deep Selective State-Space Models
por: Cirone, Nicola Muca, et al.
Publicado: (2024)

Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum
por: Cheng, Tin Sum, et al.
Publicado: (2024)

Optimizer choice matters for the emergence of Neural Collapse
por: Zhao, Jim, et al.
Publicado: (2026)

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression
por: Cheng, Tin Sum, et al.
Publicado: (2024)

Recurrent neural networks: vanishing and exploding gradients are not the end of the story
por: Zucchet, Nicolas, et al.
Publicado: (2024)

An Uncertainty Principle for Linear Recurrent Neural Networks
por: François, Alexandre, et al.
Publicado: (2025)

Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers
por: Belloni, Annalisa, et al.
Publicado: (2026)

Improved state mixing in higher-order and block diagonal linear recurrent networks
por: Dubinin, Igor, et al.
Publicado: (2026)

When, Where and Why to Average Weights?
por: Ajroldi, Niccolò, et al.
Publicado: (2025)

On the Intrinsic Dimensions of Data in Kernel Learning
por: Takhanov, Rustem
Publicado: (2026)

The informativeness of the gradient revisited
por: Takhanov, Rustem
Publicado: (2025)

Initial Guessing Bias: How Untrained Networks Favor Some Classes
por: Francazi, Emanuele, et al.
Publicado: (2023)

Conditional KRR: Injecting Unpenalized Features into Kernel Methods with Applications to Kernel Thresholding
por: Takhanov, Rustem, et al.
Publicado: (2026)

On the SDEs and Scaling Rules for Adaptive Gradient Algorithms
por: Malladi, Sadhika, et al.
Publicado: (2022)

Generator Identification for Linear SDEs with Additive and Multiplicative Noise
por: Wang, Yuanyuan, et al.
Publicado: (2023)

Optimization Guarantees for Square-Root Natural-Gradient Variational Inference
por: Kumar, Navish, et al.
Publicado: (2025)

Gradient Scalability and Taylor Surrogation of Quantum Cost Landscapes
por: Meyer, Sabri, et al.
Publicado: (2025)

Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling
por: Srećković, Teodora, et al.
Publicado: (2025)