Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914719543066624 |
|---|---|
| author | Zhang, Zeliang Jiang, Jinyang Liu, Zhuo Liang, Susan Peng, Yijie Xu, Chenliang |
| author_facet | Zhang, Zeliang Jiang, Jinyang Liu, Zhuo Liang, Susan Peng, Yijie Xu, Chenliang |
| contents | Efficient and biologically plausible alternatives to backpropagation in neural network training remain a challenge due to issues such as high computational complexity and additional assumptions about neural networks, which limit scalability to deeper networks. The likelihood ratio method offers a promising gradient estimation strategy but is constrained by significant memory consumption, especially when deploying multiple copies of data to reduce estimation variance. In this paper, we introduce an approximation technique for the likelihood ratio (LR) method to alleviate computational and memory demands in gradient estimation. By exploiting the natural parallelism during the backward pass using LR, we further provide a high-performance training strategy, which pipelines both the forward and backward pass, to make it more suitable for the computation on specialized hardware. Extensive experiments demonstrate the effectiveness of the approximation technique in neural network training. This work underscores the potential of the likelihood ratio method in achieving high-performance neural network training, suggesting avenues for further exploration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12320 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training Zhang, Zeliang Jiang, Jinyang Liu, Zhuo Liang, Susan Peng, Yijie Xu, Chenliang Machine Learning Artificial Intelligence Efficient and biologically plausible alternatives to backpropagation in neural network training remain a challenge due to issues such as high computational complexity and additional assumptions about neural networks, which limit scalability to deeper networks. The likelihood ratio method offers a promising gradient estimation strategy but is constrained by significant memory consumption, especially when deploying multiple copies of data to reduce estimation variance. In this paper, we introduce an approximation technique for the likelihood ratio (LR) method to alleviate computational and memory demands in gradient estimation. By exploiting the natural parallelism during the backward pass using LR, we further provide a high-performance training strategy, which pipelines both the forward and backward pass, to make it more suitable for the computation on specialized hardware. Extensive experiments demonstrate the effectiveness of the approximation technique in neural network training. This work underscores the potential of the likelihood ratio method in achieving high-performance neural network training, suggesting avenues for further exploration. |
| title | Approximated Likelihood Ratio: A Forward-Only and Parallel Framework for Boosting Neural Network Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.12320 |