Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.06603 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916562869420032 |
|---|---|
| author | Zhang, Yilang Li, Bingcong Giannakis, Georgios B. |
| author_facet | Zhang, Yilang Li, Bingcong Giannakis, Georgios B. |
| contents | Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06603 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm Zhang, Yilang Li, Bingcong Giannakis, Georgios B. Machine Learning Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks. |
| title | Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2501.06603 |