Improved Performance of Stochastic Gradients with Gaussian Smoothing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913579436867584 |
|---|---|
| author | Starnes, Andrew Webster, Clayton |
| author_facet | Starnes, Andrew Webster, Clayton |
| contents | This paper formalizes and analyzes Gaussian smoothing applied to two prominent optimization methods: Stochastic Gradient Descent (GSmoothSGD) and Adam (GSmoothAdam) in deep learning. By attenuating small fluctuations, Gaussian smoothing lowers the risk of gradient-based algorithms converging to poor local minima. These methods simplify the loss landscape while boosting robustness to noise and improving generalization, helping base algorithms converge more effectively to global minima. Existing approaches often rely on zero-order approximations, which increase training time due to inefficiencies in automatic differentiation. To address this, we derive Gaussian-smoothed loss functions for feedforward and convolutional networks, improving computational efficiency. Numerical experiments demonstrate the enhanced performance of our smoothing algorithms over unsmoothed counterparts, confirming the theoretical benefits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_00531 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Improved Performance of Stochastic Gradients with Gaussian Smoothing Starnes, Andrew Webster, Clayton Optimization and Control This paper formalizes and analyzes Gaussian smoothing applied to two prominent optimization methods: Stochastic Gradient Descent (GSmoothSGD) and Adam (GSmoothAdam) in deep learning. By attenuating small fluctuations, Gaussian smoothing lowers the risk of gradient-based algorithms converging to poor local minima. These methods simplify the loss landscape while boosting robustness to noise and improving generalization, helping base algorithms converge more effectively to global minima. Existing approaches often rely on zero-order approximations, which increase training time due to inefficiencies in automatic differentiation. To address this, we derive Gaussian-smoothed loss functions for feedforward and convolutional networks, improving computational efficiency. Numerical experiments demonstrate the enhanced performance of our smoothing algorithms over unsmoothed counterparts, confirming the theoretical benefits. |
| title | Improved Performance of Stochastic Gradients with Gaussian Smoothing |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2311.00531 |