Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910880887734272 |
|---|---|
| author | Clara, Gabriel Langer, Sophie Schmidt-Hieber, Johannes |
| author_facet | Clara, Gabriel Langer, Sophie Schmidt-Hieber, Johannes |
| contents | We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_10529 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model Clara, Gabriel Langer, Sophie Schmidt-Hieber, Johannes Statistics Theory Machine Learning We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expectations and covariance matrices of the iterates are derived. The results shed more light on the widely cited connection between dropout and l2-regularization in the linear model. We indicate a more subtle relationship, owing to interactions between the gradient descent dynamics and the additional randomness induced by dropout. Further, we study a simplified variant of dropout which does not have a regularizing effect and converges to the least squares estimator |
| title | Dropout Regularization Versus $\ell_2$-Penalization in the Linear Model |
| topic | Statistics Theory Machine Learning |
| url | https://arxiv.org/abs/2306.10529 |