Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910095523184640 |
|---|---|
| author | Luo, Haocheng Harandi, Mehrtash Phung, Dinh Le, Trung |
| author_facet | Luo, Haocheng Harandi, Mehrtash Phung, Dinh Le, Trung |
| contents | Sharpness-aware minimization (SAM) has emerged as a highly effective technique to improve model generalization, but its underlying principles are not fully understood. We investigate m-sharpness, where SAM performance improves monotonically as the micro-batch size for computing perturbations decreases, a phenomenon critical for distributed training yet lacking rigorous explanation. We leverage an extended Stochastic Differential Equation (SDE) framework and analyze stochastic gradient noise (SGN) to characterize the dynamics of SAM variants, including n-SAM and m-SAM. Our analysis reveals that stochastic perturbations induce an implicit variance-based sharpness regularization whose strength increases as m decreases. Motivated by this insight, we propose Reweighted SAM (RW-SAM), which employs sharpness-weighted sampling to mimic the generalization benefits of m-SAM while remaining parallelizable. Comprehensive experiments validate our theory and method.Code is available at https://github.com/RitianLuo/RW-SAM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_18001 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise Luo, Haocheng Harandi, Mehrtash Phung, Dinh Le, Trung Machine Learning Artificial Intelligence Sharpness-aware minimization (SAM) has emerged as a highly effective technique to improve model generalization, but its underlying principles are not fully understood. We investigate m-sharpness, where SAM performance improves monotonically as the micro-batch size for computing perturbations decreases, a phenomenon critical for distributed training yet lacking rigorous explanation. We leverage an extended Stochastic Differential Equation (SDE) framework and analyze stochastic gradient noise (SGN) to characterize the dynamics of SAM variants, including n-SAM and m-SAM. Our analysis reveals that stochastic perturbations induce an implicit variance-based sharpness regularization whose strength increases as m decreases. Motivated by this insight, we propose Reweighted SAM (RW-SAM), which employs sharpness-weighted sampling to mimic the generalization benefits of m-SAM while remaining parallelizable. Comprehensive experiments validate our theory and method.Code is available at https://github.com/RitianLuo/RW-SAM. |
| title | Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.18001 |