Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2206.05669 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929372120743936 |
|---|---|
| author | Li, Zhen Yang, Yunfei |
| author_facet | Li, Zhen Yang, Yunfei |
| contents | We study the uniform approximation of echo state networks with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that echo state networks with ReLU activation are universal. In this paper, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the echo state network can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU echo state networks for sufficiently regular operators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2206_05669 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Universality and approximation bounds for echo state networks with random weights Li, Zhen Yang, Yunfei Machine Learning Numerical Analysis Neural and Evolutionary Computing We study the uniform approximation of echo state networks with randomly generated internal weights. These models, in which only the readout weights are optimized during training, have made empirical success in learning dynamical systems. Recent results showed that echo state networks with ReLU activation are universal. In this paper, we give an alternative construction and prove that the universality holds for general activation functions. Specifically, our main result shows that, under certain condition on the activation function, there exists a sampling procedure for the internal weights so that the echo state network can approximate any continuous casual time-invariant operators with high probability. In particular, for ReLU activation, we give explicit construction for these sampling procedures. We also quantify the approximation error of the constructed ReLU echo state networks for sufficiently regular operators. |
| title | Universality and approximation bounds for echo state networks with random weights |
| topic | Machine Learning Numerical Analysis Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2206.05669 |