Understanding Adam Requires Better Rotation Dependent Assumptions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912690772901888 |
|---|---|
| author | Zhang, Tianyue H. Maes, Lucas Milligan, Alan Jolicoeur-Martineau, Alexia Mitliagkas, Ioannis Scieur, Damien Lacoste-Julien, Simon Guille-Escuret, Charles |
| author_facet | Zhang, Tianyue H. Maes, Lucas Milligan, Alan Jolicoeur-Martineau, Alexia Mitliagkas, Ioannis Scieur, Damien Lacoste-Julien, Simon Guille-Escuret, Charles |
| contents | Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's sensitivity to rotations of the parameter space. We observe that Adam's performance in training transformers degrades under random rotations of the parameter space, indicating a crucial sensitivity to the choice of basis in practice. This reveals that conventional rotation-invariant assumptions are insufficient to capture Adam's advantages theoretically. To better understand the rotation-dependent properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-dependent assumptions in the literature and find that they fall short in explaining Adam's behaviour across various rotation types. In contrast, we verify the orthogonality of the update as a promising indicator of Adam's basis sensitivity, suggesting it may be the key quantity for developing rotation-dependent theoretical frameworks that better explain its empirical success. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19964 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding Adam Requires Better Rotation Dependent Assumptions Zhang, Tianyue H. Maes, Lucas Milligan, Alan Jolicoeur-Martineau, Alexia Mitliagkas, Ioannis Scieur, Damien Lacoste-Julien, Simon Guille-Escuret, Charles Machine Learning Artificial Intelligence Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent (SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's sensitivity to rotations of the parameter space. We observe that Adam's performance in training transformers degrades under random rotations of the parameter space, indicating a crucial sensitivity to the choice of basis in practice. This reveals that conventional rotation-invariant assumptions are insufficient to capture Adam's advantages theoretically. To better understand the rotation-dependent properties that benefit Adam, we also identify structured rotations that preserve or even enhance its empirical performance. We then examine the rotation-dependent assumptions in the literature and find that they fall short in explaining Adam's behaviour across various rotation types. In contrast, we verify the orthogonality of the update as a promising indicator of Adam's basis sensitivity, suggesting it may be the key quantity for developing rotation-dependent theoretical frameworks that better explain its empirical success. |
| title | Understanding Adam Requires Better Rotation Dependent Assumptions |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.19964 |