Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.18153 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913908873232384 |
|---|---|
| author | Shin, Dahun Lee, Dongyeop Chung, Jinseok Lee, Namhoon |
| author_facet | Shin, Dahun Lee, Dongyeop Chung, Jinseok Lee, Namhoon |
| contents | Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend to converge to sharper minima compared to SGD. In response, we propose Sassha, a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory. In fact, this sharpness minimization scheme is crafted also to accommodate lazy Hessian updates, so as to secure efficiency besides flatness. To validate its effectiveness, we conduct a wide range of standard deep learning experiments where Sassha demonstrates its outstanding generalization performance that is comparable to, and mostly better than, other methods. We provide a comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18153 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation Shin, Dahun Lee, Dongyeop Chung, Jinseok Lee, Namhoon Machine Learning Artificial Intelligence Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend to converge to sharper minima compared to SGD. In response, we propose Sassha, a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory. In fact, this sharpness minimization scheme is crafted also to accommodate lazy Hessian updates, so as to secure efficiency besides flatness. To validate its effectiveness, we conduct a wide range of standard deep learning experiments where Sassha demonstrates its outstanding generalization performance that is comparable to, and mostly better than, other methods. We provide a comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost. |
| title | SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.18153 |