A Simplified Analysis of SGD for Linear Regression with Weight Averaging
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912438379610112 |
|---|---|
| author | Meterez, Alexandru Morwani, Depen Oncescu, Costin-Andrei Wu, Jingfeng Pehlevan, Cengiz Kakade, Sham |
| author_facet | Meterez, Alexandru Morwani, Depen Oncescu, Costin-Andrei Wu, Jingfeng Pehlevan, Cengiz Kakade, Sham |
| contents | Theoretically understanding stochastic gradient descent (SGD) in overparameterized models has led to the development of several optimization algorithms that are widely used in practice today. Recent work by~\citet{zou2021benign} provides sharp rates for SGD optimization in linear regression using constant learning rate, both with and without tail iterate averaging, based on a bias-variance decomposition of the risk. In our work, we provide a simplified analysis recovering the same bias and variance bounds provided in~\citep{zou2021benign} based on simple linear algebra tools, bypassing the requirement to manipulate operators on positive semi-definite (PSD) matrices. We believe our work makes the analysis of SGD on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling, leading to improvements in the training of realistic models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_15535 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Simplified Analysis of SGD for Linear Regression with Weight Averaging Meterez, Alexandru Morwani, Depen Oncescu, Costin-Andrei Wu, Jingfeng Pehlevan, Cengiz Kakade, Sham Machine Learning Optimization and Control Theoretically understanding stochastic gradient descent (SGD) in overparameterized models has led to the development of several optimization algorithms that are widely used in practice today. Recent work by~\citet{zou2021benign} provides sharp rates for SGD optimization in linear regression using constant learning rate, both with and without tail iterate averaging, based on a bias-variance decomposition of the risk. In our work, we provide a simplified analysis recovering the same bias and variance bounds provided in~\citep{zou2021benign} based on simple linear algebra tools, bypassing the requirement to manipulate operators on positive semi-definite (PSD) matrices. We believe our work makes the analysis of SGD on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling, leading to improvements in the training of realistic models. |
| title | A Simplified Analysis of SGD for Linear Regression with Weight Averaging |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2506.15535 |