Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matrices
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910731954290688 |
|---|---|
| author | Dexter, Gregory Boutsikas, Christos Ma, Linkai Ipsen, Ilse C. F. Drineas, Petros |
| author_facet | Dexter, Gregory Boutsikas, Christos Ma, Linkai Ipsen, Ilse C. F. Drineas, Petros |
| contents | Motivated by the popularity of stochastic rounding in the context of machine learning and the training of large-scale deep neural network models, we consider stochastic nearness rounding of real matrices $\mathbf{A}$ with many more rows than columns. We provide novel theoretical evidence, supported by extensive experimental evaluation that, with high probability, the smallest singular value of a stochastically rounded matrix is well bounded away from zero -- regardless of how close $\mathbf{A}$ is to being rank deficient and even if $\mathbf{A}$ is rank-deficient. In other words, stochastic rounding \textit{implicitly regularizes} tall and skinny matrices $\mathbf{A}$ so that the rounded version has full column rank. Our proofs leverage powerful results in random matrix theory, and the idea that stochastic rounding errors do not concentrate in low-dimensional column spaces. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12278 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matrices Dexter, Gregory Boutsikas, Christos Ma, Linkai Ipsen, Ilse C. F. Drineas, Petros Machine Learning Numerical Analysis 68W20, 65F15, 65F22, 65G50, 15A18, 15A42 Motivated by the popularity of stochastic rounding in the context of machine learning and the training of large-scale deep neural network models, we consider stochastic nearness rounding of real matrices $\mathbf{A}$ with many more rows than columns. We provide novel theoretical evidence, supported by extensive experimental evaluation that, with high probability, the smallest singular value of a stochastically rounded matrix is well bounded away from zero -- regardless of how close $\mathbf{A}$ is to being rank deficient and even if $\mathbf{A}$ is rank-deficient. In other words, stochastic rounding \textit{implicitly regularizes} tall and skinny matrices $\mathbf{A}$ so that the rounded version has full column rank. Our proofs leverage powerful results in random matrix theory, and the idea that stochastic rounding errors do not concentrate in low-dimensional column spaces. |
| title | Stochastic Rounding Implicitly Regularizes Tall-and-Thin Matrices |
| topic | Machine Learning Numerical Analysis 68W20, 65F15, 65F22, 65G50, 15A18, 15A42 |
| url | https://arxiv.org/abs/2403.12278 |