Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.13765 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912215867588608 |
|---|---|
| author | Gibson, Travis E. Acharya, Sawal Parashar, Anjali Gaudio, Joseph E. Annaswamy, Anurdha M. |
| author_facet | Gibson, Travis E. Acharya, Sawal Parashar, Anjali Gaudio, Joseph E. Annaswamy, Anurdha M. |
| contents | Gradient based optimization algorithms deployed in Machine Learning (ML) applications are often analyzed and compared by their convergence rates or regret bounds. While these rates and bounds convey valuable information they don't always directly translate to stability guarantees. Stability and similar concepts, like robustness, will become ever more important as we move towards deploying models in real-time and safety critical systems. In this work we build upon the results in Gaudio et al. 2021 and Moreu & Annaswamy 2022 for gradient descent with second order dynamics when applied to explicitly time varying cost functions and provide more general stability guarantees. These more general results can aid in the design and certification of these optimization schemes so as to help ensure safe and reliable deployment for real-time learning applications. We also hope that the techniques provided here will stimulate and cross-fertilize the analysis that occurs on the same algorithms from the online learning and stochastic optimization communities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13765 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the stability of gradient descent with second order dynamics for time-varying cost functions Gibson, Travis E. Acharya, Sawal Parashar, Anjali Gaudio, Joseph E. Annaswamy, Anurdha M. Machine Learning Optimization and Control Gradient based optimization algorithms deployed in Machine Learning (ML) applications are often analyzed and compared by their convergence rates or regret bounds. While these rates and bounds convey valuable information they don't always directly translate to stability guarantees. Stability and similar concepts, like robustness, will become ever more important as we move towards deploying models in real-time and safety critical systems. In this work we build upon the results in Gaudio et al. 2021 and Moreu & Annaswamy 2022 for gradient descent with second order dynamics when applied to explicitly time varying cost functions and provide more general stability guarantees. These more general results can aid in the design and certification of these optimization schemes so as to help ensure safe and reliable deployment for real-time learning applications. We also hope that the techniques provided here will stimulate and cross-fertilize the analysis that occurs on the same algorithms from the online learning and stochastic optimization communities. |
| title | On the stability of gradient descent with second order dynamics for time-varying cost functions |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2405.13765 |