Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916959414648832 |
|---|---|
| author | Tsilivis, Nikolaos Gronich, Eitan Kempe, Julia Vardi, Gal |
| author_facet | Tsilivis, Nikolaos Gronich, Eitan Kempe, Julia Vardi, Gal |
| contents | We study the implicit bias of the general family of steepest descent algorithms with infinitesimal learning rate in deep homogeneous neural networks. We show that: (a) an algorithm-dependent geometric margin starts increasing once the networks reach perfect training accuracy, and (b) any limit point of the training trajectory corresponds to a KKT point of the corresponding margin-maximization problem. We experimentally zoom into the trajectories of neural networks optimized with various steepest descent algorithms, highlighting connections to the implicit bias of popular adaptive methods (Adam and Shampoo). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22069 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks Tsilivis, Nikolaos Gronich, Eitan Kempe, Julia Vardi, Gal Machine Learning We study the implicit bias of the general family of steepest descent algorithms with infinitesimal learning rate in deep homogeneous neural networks. We show that: (a) an algorithm-dependent geometric margin starts increasing once the networks reach perfect training accuracy, and (b) any limit point of the training trajectory corresponds to a KKT point of the corresponding margin-maximization problem. We experimentally zoom into the trajectories of neural networks optimized with various steepest descent algorithms, highlighting connections to the implicit bias of popular adaptive methods (Adam and Shampoo). |
| title | Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.22069 |