Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2304.13328 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929218744483840 |
|---|---|
| author | Le, Tam |
| author_facet | Le, Tam |
| contents | Motivated by the conspicuous use of momentum-based algorithms in deep learning, we study a nonsmooth nonconvex stochastic heavy ball method and show its convergence. Our approach builds upon semialgebraic (definable) assumptions commonly met in practical situations and combines a nonsmooth calculus with a differential inclusion method. Additionally, we provide general conditions for the sample distribution to ensure the convergence of the objective function. Our results are general enough to justify the use of subgradient sampling in modern implementations that heuristically apply rules of differential calculus on nonsmooth functions, such as backpropagation or implicit differentiation. As for the stochastic subgradient method, our analysis highlights that subgradient sampling can make the stochastic heavy ball method converge to artificial critical points. Thanks to the semialgebraic setting, we address this concern showing that these artifacts are almost surely avoided when initializations are randomized, leading the method to converge to Clarke critical points. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_13328 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Nonsmooth nonconvex stochastic heavy ball Le, Tam Optimization and Control Motivated by the conspicuous use of momentum-based algorithms in deep learning, we study a nonsmooth nonconvex stochastic heavy ball method and show its convergence. Our approach builds upon semialgebraic (definable) assumptions commonly met in practical situations and combines a nonsmooth calculus with a differential inclusion method. Additionally, we provide general conditions for the sample distribution to ensure the convergence of the objective function. Our results are general enough to justify the use of subgradient sampling in modern implementations that heuristically apply rules of differential calculus on nonsmooth functions, such as backpropagation or implicit differentiation. As for the stochastic subgradient method, our analysis highlights that subgradient sampling can make the stochastic heavy ball method converge to artificial critical points. Thanks to the semialgebraic setting, we address this concern showing that these artifacts are almost surely avoided when initializations are randomized, leading the method to converge to Clarke critical points. |
| title | Nonsmooth nonconvex stochastic heavy ball |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2304.13328 |