Activation Bottleneck: Sigmoidal Neural Networks Cannot Forecast a Straight Line
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913376952647680 |
|---|---|
| author | Toller, Maximilian Hussain, Hussain Geiger, Bernhard C |
| author_facet | Toller, Maximilian Hussain, Hussain Geiger, Bernhard C |
| contents | A neural network has an activation bottleneck if one of its hidden layers has a bounded image. We show that networks with an activation bottleneck cannot forecast unbounded sequences such as straight lines, random walks, or any sequence with a trend: The difference between prediction and ground truth becomes arbitrary large, regardless of the training procedure. Widely-used neural network architectures such as LSTM and GRU suffer from this limitation. In our analysis, we characterize activation bottlenecks and explain why they prevent sigmoidal networks from learning unbounded sequences. We experimentally validate our findings and discuss modifications to network architectures which mitigate the effects of activation bottlenecks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02146 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Activation Bottleneck: Sigmoidal Neural Networks Cannot Forecast a Straight Line Toller, Maximilian Hussain, Hussain Geiger, Bernhard C Machine Learning A neural network has an activation bottleneck if one of its hidden layers has a bounded image. We show that networks with an activation bottleneck cannot forecast unbounded sequences such as straight lines, random walks, or any sequence with a trend: The difference between prediction and ground truth becomes arbitrary large, regardless of the training procedure. Widely-used neural network architectures such as LSTM and GRU suffer from this limitation. In our analysis, we characterize activation bottlenecks and explain why they prevent sigmoidal networks from learning unbounded sequences. We experimentally validate our findings and discuss modifications to network architectures which mitigate the effects of activation bottlenecks. |
| title | Activation Bottleneck: Sigmoidal Neural Networks Cannot Forecast a Straight Line |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2406.02146 |