There is a Singularity in the Loss Landscape
Fuente:
arXiv
Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Preprint |
| Publié: |
2022
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914877880139776 |
|---|---|
| author | Lowell, Mark |
| author_facet | Lowell, Mark |
| contents | Despite the widespread adoption of neural networks, their training dynamics remain poorly understood. We show experimentally that as the size of the dataset increases, a point forms where the magnitude of the gradient of the loss becomes unbounded. Gradient descent rapidly brings the network close to this singularity in parameter space, and further training takes place near it. This singularity explains a variety of phenomena recently observed in the Hessian of neural network loss functions, such as training on the edge of stability and the concentration of the gradient in a top subspace. Once the network approaches the singularity, the top subspace contributes little to learning, even though it constitutes the majority of the gradient. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2201_06964 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | There is a Singularity in the Loss Landscape Lowell, Mark Machine Learning Artificial Intelligence Despite the widespread adoption of neural networks, their training dynamics remain poorly understood. We show experimentally that as the size of the dataset increases, a point forms where the magnitude of the gradient of the loss becomes unbounded. Gradient descent rapidly brings the network close to this singularity in parameter space, and further training takes place near it. This singularity explains a variety of phenomena recently observed in the Hessian of neural network loss functions, such as training on the edge of stability and the concentration of the gradient in a top subspace. Once the network approaches the singularity, the top subspace contributes little to learning, even though it constitutes the majority of the gradient. |
| title | There is a Singularity in the Loss Landscape |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2201.06964 |