Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918398102863872 |
|---|---|
| author | Dereich, Steffen Do, Thang Jentzen, Arnulf |
| author_facet | Dereich, Steffen Do, Thang Jentzen, Arnulf |
| contents | The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_18899 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method Dereich, Steffen Do, Thang Jentzen, Arnulf Machine Learning Optimization and Control 68T05, 90C25, 65K05, 65K10, 60H35 I.2.0; G.3; G.1.6; F.2.1 The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence (AI) systems. Despite its groundbreaking success in the training of AI systems, it still remains an open research problem to provide a complete error analysis of Adam, not only for optimizing DNNs but even when applied to strongly convex stochastic optimization problems (SOPs). Previous error analysis results for strongly convex SOPs in the literature provide conditional convergence analyses that rely on the assumption that Adam does not diverge to infinity but remains uniformly bounded. It is the key contribution of this work to establish uniform a priori bounds for Adam and, thereby, to provide -- for the first time -- an unconditional error analysis for Adam for a large class of strongly convex SOPs. |
| title | Uniform a priori bounds and error analysis for the Adam stochastic gradient descent optimization method |
| topic | Machine Learning Optimization and Control 68T05, 90C25, 65K05, 65K10, 60H35 I.2.0; G.3; G.1.6; F.2.1 |
| url | https://arxiv.org/abs/2603.18899 |