Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912907265048576 |
|---|---|
| author | Bakong, Killian Massoulié, Laurent Oyallon, Edouard Scaman, Kevin |
| author_facet | Bakong, Killian Massoulié, Laurent Oyallon, Edouard Scaman, Kevin |
| contents | In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products by randomized, unbiased approximations thereof during backpropagation. We provide a theoretical analysis of the trade-off between the number of epochs needed to achieve a target precision and the cost reduction for each epoch. We then identify specific unbiased estimates of vector-jacobian products for which we establish desirable optimality properties of minimal variance under sparsity constraints. Finally we provide in-depth experiments on multi-layer perceptrons, BagNets and Visual Transfomers architectures. These validate our theoretical results, and confirm the potential of our proposed unbiased randomized backpropagation approach for reducing the cost of deep learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14701 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation Bakong, Killian Massoulié, Laurent Oyallon, Edouard Scaman, Kevin Machine Learning In this work we introduce methods to reduce the computational and memory costs of training deep neural networks. Our approach consists in replacing exact vector-jacobian products by randomized, unbiased approximations thereof during backpropagation. We provide a theoretical analysis of the trade-off between the number of epochs needed to achieve a target precision and the cost reduction for each epoch. We then identify specific unbiased estimates of vector-jacobian products for which we establish desirable optimality properties of minimal variance under sparsity constraints. Finally we provide in-depth experiments on multi-layer perceptrons, BagNets and Visual Transfomers architectures. These validate our theoretical results, and confirm the potential of our proposed unbiased randomized backpropagation approach for reducing the cost of deep learning. |
| title | Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.14701 |