Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912258190213120 |
|---|---|
| author | Veiga, Rodrigo Remizova, Anastasia Macris, Nicolas |
| author_facet | Veiga, Rodrigo Remizova, Anastasia Macris, Nicolas |
| contents | We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing the difference between test risk curves of pure gradient and stochastic gradient flows. We apply the general theory to a simple model of weak features, which displays the double descent phenomenon, and explicitly compute the corrections brought about by the added stochastic term in the dynamics, as a function of time and model parameters. The analytical results are compared to simulations of discrete-time stochastic gradient descent and show good agreement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07626 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features Veiga, Rodrigo Remizova, Anastasia Macris, Nicolas Machine Learning Disordered Systems and Neural Networks We investigate the test risk of continuous-time stochastic gradient flow dynamics in learning theory. Using a path integral formulation we provide, in the regime of a small learning rate, a general formula for computing the difference between test risk curves of pure gradient and stochastic gradient flows. We apply the general theory to a simple model of weak features, which displays the double descent phenomenon, and explicitly compute the corrections brought about by the added stochastic term in the dynamics, as a function of time and model parameters. The analytical results are compared to simulations of discrete-time stochastic gradient descent and show good agreement. |
| title | Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2402.07626 |