Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913871654027264 |
|---|---|
| author | Gaash, Ofir Levy, Kfir Yehuda Carmon, Yair |
| author_facet | Gaash, Ofir Levy, Kfir Yehuda Carmon, Yair |
| contents | We study stochastic gradient descent (SGD) with gradient clipping on convex functions under a generalized smoothness assumption called $(L_0,L_1)$-smoothness. Using gradient clipping, we establish a high probability convergence rate that matches the SGD rate in the $L$ smooth case up to polylogarithmic factors and additive terms. We also propose a variation of adaptive SGD with gradient clipping, which achieves the same guarantee. We perform empirical experiments to examine our theory and algorithmic choices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16492 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions Gaash, Ofir Levy, Kfir Yehuda Carmon, Yair Optimization and Control We study stochastic gradient descent (SGD) with gradient clipping on convex functions under a generalized smoothness assumption called $(L_0,L_1)$-smoothness. Using gradient clipping, we establish a high probability convergence rate that matches the SGD rate in the $L$ smooth case up to polylogarithmic factors and additive terms. We also propose a variation of adaptive SGD with gradient clipping, which achieves the same guarantee. We perform empirical experiments to examine our theory and algorithmic choices. |
| title | Convergence of Clipped SGD on Convex $(L_0,L_1)$-Smooth Functions |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2502.16492 |