Forward Only Learning for Orthogonal Neural Networks of any Depth
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908730696663040 |
|---|---|
| author | Caillon, Paul Colagrande, Alex Fagnou, Erwan Delattre, Blaise Allauzen, Alexandre |
| author_facet | Caillon, Paul Colagrande, Alex Fagnou, Erwan Delattre, Blaise Allauzen, Alexandre |
| contents | Backpropagation is still the de facto algorithm used today to
train neural networks.
With the exponential growth of recent architectures, the
computational cost of this algorithm also becomes a burden. The
recent PEPITA and forward-only frameworks have proposed promising
alternatives, but they failed to scale up to a handful of hidden
layers, yet limiting their use.
In this paper, we first analyze theoretically the main limitations of
these approaches. It allows us the design of a forward-only
algorithm, which is equivalent to backpropagation under the linear
and orthogonal assumptions. By relaxing the linear assumption, we
then introduce FOTON (Forward-Only Training of Orthogonal Networks)
that bridges the gap with the backpropagation
algorithm. Experimental results show that it outperforms PEPITA,
enabling us to train neural networks of any depth, without the need
for a backward pass.
Moreover its performance on convolutional networks clearly opens up avenues for its application to more
advanced architectures. The code is open-sourced at https://github.com/p0lcAi/FOTON . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_20668 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Forward Only Learning for Orthogonal Neural Networks of any Depth Caillon, Paul Colagrande, Alex Fagnou, Erwan Delattre, Blaise Allauzen, Alexandre Machine Learning Artificial Intelligence Backpropagation is still the de facto algorithm used today to train neural networks. With the exponential growth of recent architectures, the computational cost of this algorithm also becomes a burden. The recent PEPITA and forward-only frameworks have proposed promising alternatives, but they failed to scale up to a handful of hidden layers, yet limiting their use. In this paper, we first analyze theoretically the main limitations of these approaches. It allows us the design of a forward-only algorithm, which is equivalent to backpropagation under the linear and orthogonal assumptions. By relaxing the linear assumption, we then introduce FOTON (Forward-Only Training of Orthogonal Networks) that bridges the gap with the backpropagation algorithm. Experimental results show that it outperforms PEPITA, enabling us to train neural networks of any depth, without the need for a backward pass. Moreover its performance on convolutional networks clearly opens up avenues for its application to more advanced architectures. The code is open-sourced at https://github.com/p0lcAi/FOTON . |
| title | Forward Only Learning for Orthogonal Neural Networks of any Depth |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.20668 |