Stochastic gradient descent for streaming linear and rectified linear systems with adversarial corruptions
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909458932695040 |
|---|---|
| author | Jeong, Halyun Needell, Deanna Rebrova, Elizaveta |
| author_facet | Jeong, Halyun Needell, Deanna Rebrova, Elizaveta |
| contents | We propose SGD-exp, a stochastic gradient descent approach for linear and ReLU regressions under Massart noise (adversarial semi-random corruption model) for the fully streaming setting. We show novel nearly linear convergence guarantees of SGD-exp to the true parameter with up to $50\%$ Massart corruption rate, and with any corruption rate in the case of symmetric oblivious corruptions. This is the first convergence guarantee result for robust ReLU regression in the streaming setting, and it shows the improved convergence rate over previous robust methods for $L_1$ linear regression due to a choice of an exponentially decaying step size, known for its efficiency in practice. Our analysis is based on the drift analysis of a discrete stochastic process, which could also be interesting on its own. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01204 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Stochastic gradient descent for streaming linear and rectified linear systems with adversarial corruptions Jeong, Halyun Needell, Deanna Rebrova, Elizaveta Machine Learning Numerical Analysis 65F10, 60-XX We propose SGD-exp, a stochastic gradient descent approach for linear and ReLU regressions under Massart noise (adversarial semi-random corruption model) for the fully streaming setting. We show novel nearly linear convergence guarantees of SGD-exp to the true parameter with up to $50\%$ Massart corruption rate, and with any corruption rate in the case of symmetric oblivious corruptions. This is the first convergence guarantee result for robust ReLU regression in the streaming setting, and it shows the improved convergence rate over previous robust methods for $L_1$ linear regression due to a choice of an exponentially decaying step size, known for its efficiency in practice. Our analysis is based on the drift analysis of a discrete stochastic process, which could also be interesting on its own. |
| title | Stochastic gradient descent for streaming linear and rectified linear systems with adversarial corruptions |
| topic | Machine Learning Numerical Analysis 65F10, 60-XX |
| url | https://arxiv.org/abs/2403.01204 |