Noise-induced degeneration in online learning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2020
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913349230395392 |
|---|---|
| author | Sato, Yuzuru Tsutsui, Daiji Fujiwara, Akio |
| author_facet | Sato, Yuzuru Tsutsui, Daiji Fujiwara, Akio |
| contents | In order to elucidate the plateau phenomena caused by vanishing gradient, we herein analyse stability of stochastic gradient descent near degenerated subspaces in a multi-layer perceptron. In stochastic gradient descent for Fukumizu-Amari model, which is the minimal multi-layer perceptron showing non-trivial plateau phenomena, we show that (1) attracting regions exist in multiply degenerated subspaces, (2) a strong plateau phenomenon emerges as a noise-induced synchronisation, which is not observed in deterministic gradient descent, (3) an optimal fluctuation exists to minimise the escape time from the degenerated subspace. The noise-induced degeneration observed herein is expected to be found in a broad class of machine learning via neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2008_10498 |
| institution | arXiv |
| publishDate | 2020 |
| record_format | arxiv |
| spellingShingle | Noise-induced degeneration in online learning Sato, Yuzuru Tsutsui, Daiji Fujiwara, Akio Adaptation and Self-Organizing Systems Machine Learning Dynamical Systems Chaotic Dynamics In order to elucidate the plateau phenomena caused by vanishing gradient, we herein analyse stability of stochastic gradient descent near degenerated subspaces in a multi-layer perceptron. In stochastic gradient descent for Fukumizu-Amari model, which is the minimal multi-layer perceptron showing non-trivial plateau phenomena, we show that (1) attracting regions exist in multiply degenerated subspaces, (2) a strong plateau phenomenon emerges as a noise-induced synchronisation, which is not observed in deterministic gradient descent, (3) an optimal fluctuation exists to minimise the escape time from the degenerated subspace. The noise-induced degeneration observed herein is expected to be found in a broad class of machine learning via neural networks. |
| title | Noise-induced degeneration in online learning |
| topic | Adaptation and Self-Organizing Systems Machine Learning Dynamical Systems Chaotic Dynamics |
| url | https://arxiv.org/abs/2008.10498 |