SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913908873232384 |
|---|---|
| author | Shin, Dahun Lee, Dongyeop Chung, Jinseok Lee, Namhoon |
| author_facet | Shin, Dahun Lee, Dongyeop Chung, Jinseok Lee, Namhoon |
| contents | Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend to converge to sharper minima compared to SGD. In response, we propose Sassha, a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory. In fact, this sharpness minimization scheme is crafted also to accommodate lazy Hessian updates, so as to secure efficiency besides flatness. To validate its effectiveness, we conduct a wide range of standard deep learning experiments where Sassha demonstrates its outstanding generalization performance that is comparable to, and mostly better than, other methods. We provide a comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18153 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation Shin, Dahun Lee, Dongyeop Chung, Jinseok Lee, Namhoon Machine Learning Artificial Intelligence Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend to converge to sharper minima compared to SGD. In response, we propose Sassha, a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory. In fact, this sharpness minimization scheme is crafted also to accommodate lazy Hessian updates, so as to secure efficiency besides flatness. To validate its effectiveness, we conduct a wide range of standard deep learning experiments where Sassha demonstrates its outstanding generalization performance that is comparable to, and mostly better than, other methods. We provide a comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost. |
| title | SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.18153 |