Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913773171769344 |
|---|---|
| author | Gower, Robert M. Garrigos, Guillaume Loizou, Nicolas Oikonomou, Dimitris Mishchenko, Konstantin Schaipp, Fabian |
| author_facet | Gower, Robert M. Garrigos, Guillaume Loizou, Nicolas Oikonomou, Dimitris Mishchenko, Konstantin Schaipp, Fabian |
| contents | We provide a general convergence theorem of an idealized stochastic Polyak step size called SPS$^*$. Besides convexity, we only assume a local expected gradient bound, that includes locally smooth and locally Lipschitz losses as special cases. We refer to SPS$^*$ as idealized because it requires access to the loss for every training batch evaluated at a solution. It is also ideal, in that it achieves the optimal lower bound for globally Lipschitz function, and is the first Polyak step size to have an $O(1/\sqrt{t})$ anytime convergence in the smooth setting. We show how to combine SPS$^*$ with momentum to achieve the same favorable rates for the last iterate. We conclude with several experiments to validate our theory, and a more practical setting showing how we can distill a teacher GPT-2 model into a smaller student model without any hyperparameter tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01898 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation Gower, Robert M. Garrigos, Guillaume Loizou, Nicolas Oikonomou, Dimitris Mishchenko, Konstantin Schaipp, Fabian Machine Learning 90C53, 74S60, 90C06, 62L20, 68W20, 15B52, 65Y20, 68W40 G.1.6 We provide a general convergence theorem of an idealized stochastic Polyak step size called SPS$^*$. Besides convexity, we only assume a local expected gradient bound, that includes locally smooth and locally Lipschitz losses as special cases. We refer to SPS$^*$ as idealized because it requires access to the loss for every training batch evaluated at a solution. It is also ideal, in that it achieves the optimal lower bound for globally Lipschitz function, and is the first Polyak step size to have an $O(1/\sqrt{t})$ anytime convergence in the smooth setting. We show how to combine SPS$^*$ with momentum to achieve the same favorable rates for the last iterate. We conclude with several experiments to validate our theory, and a more practical setting showing how we can distill a teacher GPT-2 model into a smaller student model without any hyperparameter tuning. |
| title | Analysis of an Idealized Stochastic Polyak Method and its Application to Black-Box Model Distillation |
| topic | Machine Learning 90C53, 74S60, 90C06, 62L20, 68W20, 15B52, 65Y20, 68W40 G.1.6 |
| url | https://arxiv.org/abs/2504.01898 |