Exploring Landscapes for Better Minima along Valleys
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915588380557312 |
|---|---|
| author | Zhao, Tong Li, Jiacheng Zhou, Yuanchang Tan, Guangming Jia, Weile |
| author_facet | Zhao, Tong Li, Jiacheng Zhou, Yuanchang Tan, Guangming Jia, Weile |
| contents | Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such a point is the lowest or provides the best generalization. To address this, we propose an adaptor "E" for gradient-based optimizers. The adapted optimizer tends to continue exploring along landscape valleys (areas with low and nearly identical losses) in order to search for potentially better local minima even after reaching a local minimum. This approach increases the likelihood of finding a lower and flatter local minimum, which is often associated with better generalization. We also provide a proof of convergence for the adapted optimizers in both convex and non-convex scenarios for completeness. Finally, we demonstrate their effectiveness in an important but notoriously difficult training scenario, large-batch training, where Lamb is the benchmark optimizer. Our testing results show that the adapted Lamb, ALTO, increases the test accuracy (generalization) of the current state-of-the-art optimizer by an average of 2.5% across a variety of large-batch training tasks. This work potentially opens a new research direction in the design of optimization algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_27153 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring Landscapes for Better Minima along Valleys Zhao, Tong Li, Jiacheng Zhou, Yuanchang Tan, Guangming Jia, Weile Machine Learning Artificial Intelligence Optimization and Control 65K05, 65K10 (Primary) 49K05, 49J15, 90C26, 62F10 (Secondary) D.1.2; D.2.1; D.2.5 Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such a point is the lowest or provides the best generalization. To address this, we propose an adaptor "E" for gradient-based optimizers. The adapted optimizer tends to continue exploring along landscape valleys (areas with low and nearly identical losses) in order to search for potentially better local minima even after reaching a local minimum. This approach increases the likelihood of finding a lower and flatter local minimum, which is often associated with better generalization. We also provide a proof of convergence for the adapted optimizers in both convex and non-convex scenarios for completeness. Finally, we demonstrate their effectiveness in an important but notoriously difficult training scenario, large-batch training, where Lamb is the benchmark optimizer. Our testing results show that the adapted Lamb, ALTO, increases the test accuracy (generalization) of the current state-of-the-art optimizer by an average of 2.5% across a variety of large-batch training tasks. This work potentially opens a new research direction in the design of optimization algorithms. |
| title | Exploring Landscapes for Better Minima along Valleys |
| topic | Machine Learning Artificial Intelligence Optimization and Control 65K05, 65K10 (Primary) 49K05, 49J15, 90C26, 62F10 (Secondary) D.1.2; D.2.1; D.2.5 |
| url | https://arxiv.org/abs/2510.27153 |