Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.17567 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929605939560448 |
|---|---|
| author | Dexheimer, Niklas Schmidt-Hieber, Johannes |
| author_facet | Dexheimer, Niklas Schmidt-Hieber, Johannes |
| contents | Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linear model with $d$ parameters, previous work has found that the prediction error of FGD is, however, by a factor $d$ slower than the prediction error of stochastic gradient descent (SGD). In this paper we show that by computing $\ell$ FGD steps based on each training sample, this suboptimality factor becomes $d/(\ell \wedge d)$ and thus the suboptimality of the rate disappears if $\ell \gtrsim d.$ We also show that FGD with repeated sampling can adapt to low-dimensional structure in the input distribution. The main mathematical challenge lies in controlling the dependencies arising from the repeated sampling process. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17567 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling Dexheimer, Niklas Schmidt-Hieber, Johannes Statistics Theory Machine Learning Neural and Evolutionary Computing 62L20, 62J05 Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linear model with $d$ parameters, previous work has found that the prediction error of FGD is, however, by a factor $d$ slower than the prediction error of stochastic gradient descent (SGD). In this paper we show that by computing $\ell$ FGD steps based on each training sample, this suboptimality factor becomes $d/(\ell \wedge d)$ and thus the suboptimality of the rate disappears if $\ell \gtrsim d.$ We also show that FGD with repeated sampling can adapt to low-dimensional structure in the input distribution. The main mathematical challenge lies in controlling the dependencies arising from the repeated sampling process. |
| title | Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling |
| topic | Statistics Theory Machine Learning Neural and Evolutionary Computing 62L20, 62J05 |
| url | https://arxiv.org/abs/2411.17567 |