Why Adam Works Better with $β_1 = β_2$: The Missing Gradient Scale Invariance Principle
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911668505673728 |
|---|---|
| author | Fernández-Hernández, Alberto Pérez-Corral, Cristian Mestre, Jose I. Dolz, Manuel F. Quintana-Ortí, Enrique S. |
| author_facet | Fernández-Hernández, Alberto Pérez-Corral, Cristian Mestre, Jose I. Dolz, Manuel F. Quintana-Ortí, Enrique S. |
| contents | Adam has been at the core of large-scale training for almost a decade, yet a simple empirical fact remains unaccounted for: both validation scores and the qualitative behaviour of the training runs improve when the momentum parameters satisfy $β_{1}=β_{2}$. Some recent studies have reported this pattern, but there is still no explanation for why this choice helps. We show that this choice is closely tied to a structural property that we refer to as \textit{gradient scale invariance}. We formalize this notion and prove that Adam becomes gradient scale invariant of first order if and only if $β_{1}=β_{2}$. This perspective places the balanced regime of Adam in direct alignment with the design principles underlying several recent optimizers that explicitly enforce scale-robust updates. The theory is supported by experiments across vision and language tasks, and across different architectural families, in which rescaling the gradient has a markedly smoother effect on the update when $β_{1}=β_{2}$. Overall, our results offer a coherent explanation for an open question in the behavior of Adam and provide a simple principle that helps guide the design of future optimizers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21739 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Why Adam Works Better with $β_1 = β_2$: The Missing Gradient Scale Invariance Principle Fernández-Hernández, Alberto Pérez-Corral, Cristian Mestre, Jose I. Dolz, Manuel F. Quintana-Ortí, Enrique S. Machine Learning Artificial Intelligence Adam has been at the core of large-scale training for almost a decade, yet a simple empirical fact remains unaccounted for: both validation scores and the qualitative behaviour of the training runs improve when the momentum parameters satisfy $β_{1}=β_{2}$. Some recent studies have reported this pattern, but there is still no explanation for why this choice helps. We show that this choice is closely tied to a structural property that we refer to as \textit{gradient scale invariance}. We formalize this notion and prove that Adam becomes gradient scale invariant of first order if and only if $β_{1}=β_{2}$. This perspective places the balanced regime of Adam in direct alignment with the design principles underlying several recent optimizers that explicitly enforce scale-robust updates. The theory is supported by experiments across vision and language tasks, and across different architectural families, in which rescaling the gradient has a markedly smoother effect on the update when $β_{1}=β_{2}$. Overall, our results offer a coherent explanation for an open question in the behavior of Adam and provide a simple principle that helps guide the design of future optimizers. |
| title | Why Adam Works Better with $β_1 = β_2$: The Missing Gradient Scale Invariance Principle |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.21739 |