Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910981916983296 |
|---|---|
| author | Li, Sicong Xu, Qianqian Yang, Zhiyong Wang, Zitai Zhang, Linchao Cao, Xiaochun Huang, Qingming |
| author_facet | Li, Sicong Xu, Qianqian Yang, Zhiyong Wang, Zitai Zhang, Linchao Cao, Xiaochun Huang, Qingming |
| contents | Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult. Recent methods resorted to long-tail variants of Sharpness-Aware Minimization (SAM), such as ImbSAM and CC-SAM, to improve generalization by flattening the loss landscape. However, these attempts face a trade-off between computational efficiency and control over the loss landscape. On the one hand, ImbSAM is efficient but offers only coarse control as it excludes head classes from the SAM process. On the other hand, CC-SAM provides fine-grained control through class-dependent perturbations but at the cost of efficiency due to multiple backpropagations. Seeing this dilemma, we introduce Focal-SAM, which assigns different penalties to class-wise sharpness, achieving fine-grained control without extra backpropagations, thus maintaining efficiency. Furthermore, we theoretically analyze Focal-SAM's generalization ability and derive a sharper generalization bound. Extensive experiments on both traditional and foundation models validate the effectiveness of Focal-SAM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_01660 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification Li, Sicong Xu, Qianqian Yang, Zhiyong Wang, Zitai Zhang, Linchao Cao, Xiaochun Huang, Qingming Machine Learning Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult. Recent methods resorted to long-tail variants of Sharpness-Aware Minimization (SAM), such as ImbSAM and CC-SAM, to improve generalization by flattening the loss landscape. However, these attempts face a trade-off between computational efficiency and control over the loss landscape. On the one hand, ImbSAM is efficient but offers only coarse control as it excludes head classes from the SAM process. On the other hand, CC-SAM provides fine-grained control through class-dependent perturbations but at the cost of efficiency due to multiple backpropagations. Seeing this dilemma, we introduce Focal-SAM, which assigns different penalties to class-wise sharpness, achieving fine-grained control without extra backpropagations, thus maintaining efficiency. Furthermore, we theoretically analyze Focal-SAM's generalization ability and derive a sharper generalization bound. Extensive experiments on both traditional and foundation models validate the effectiveness of Focal-SAM. |
| title | Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.01660 |