AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908381924556800 |
|---|---|
| author | Surjanovic, Nikola Bouchard-Côté, Alexandre Campbell, Trevor |
| author_facet | Surjanovic, Nikola Bouchard-Côté, Alexandre Campbell, Trevor |
| contents | The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21651 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent Surjanovic, Nikola Bouchard-Côté, Alexandre Campbell, Trevor Machine Learning Optimization and Control Computation The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks. |
| title | AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent |
| topic | Machine Learning Optimization and Control Computation |
| url | https://arxiv.org/abs/2505.21651 |