Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.14168 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914664047181824 |
|---|---|
| author | Shukla, Khemraj Shin, Yeonjong |
| author_facet | Shukla, Khemraj Shin, Yeonjong |
| contents | We present a randomized forward mode gradient (RFG) as an alternative to backpropagation. RFG is a random estimator for the gradient that is constructed based on the directional derivative along a random vector. The forward mode automatic differentiation (AD) provides an efficient computation of RFG. The probability distribution of the random vector determines the statistical properties of RFG. Through the second moment analysis, we found that the distribution with the smallest kurtosis yields the smallest expected relative squared error. By replacing gradient with RFG, a class of RFG-based optimization algorithms is obtained. By focusing on gradient descent (GD) and Polyak's heavy ball (PHB) methods, we present a convergence analysis of RFG-based optimization algorithms for quadratic functions. Computational experiments are presented to demonstrate the performance of the proposed algorithms and verify the theoretical findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_14168 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Randomized Forward Mode of Automatic Differentiation For Optimization Algorithms Shukla, Khemraj Shin, Yeonjong Optimization and Control Artificial Intelligence Machine Learning 65K05, 65B99, 65Y20 We present a randomized forward mode gradient (RFG) as an alternative to backpropagation. RFG is a random estimator for the gradient that is constructed based on the directional derivative along a random vector. The forward mode automatic differentiation (AD) provides an efficient computation of RFG. The probability distribution of the random vector determines the statistical properties of RFG. Through the second moment analysis, we found that the distribution with the smallest kurtosis yields the smallest expected relative squared error. By replacing gradient with RFG, a class of RFG-based optimization algorithms is obtained. By focusing on gradient descent (GD) and Polyak's heavy ball (PHB) methods, we present a convergence analysis of RFG-based optimization algorithms for quadratic functions. Computational experiments are presented to demonstrate the performance of the proposed algorithms and verify the theoretical findings. |
| title | Randomized Forward Mode of Automatic Differentiation For Optimization Algorithms |
| topic | Optimization and Control Artificial Intelligence Machine Learning 65K05, 65B99, 65Y20 |
| url | https://arxiv.org/abs/2310.14168 |