Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Data
Fuente:
arXiv
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915462284050432 |
|---|---|
| author | Siegel, Jonathan W. |
| author_facet | Siegel, Jonathan W. |
| contents | We study the interpolation power of deep ReLU neural networks. Specifically, we consider the question of how efficiently, in terms of the number of parameters, deep ReLU networks can interpolate values at $N$ datapoints in the unit ball which are separated by a distance $δ$. We show that $Ω(N)$ parameters are required in the regime where $δ$ is exponentially small in $N$, which gives the sharp result in this regime since $O(N)$ parameters are always sufficient. This also shows that the bit-extraction technique used to prove lower bounds on the VC dimension cannot be applied to irregularly spaced datapoints. Finally, as an application we give a lower bound on the approximation rates that deep ReLU neural networks can achieve for Sobolev spaces at the embedding endpoint. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_00834 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Data Siegel, Jonathan W. Machine Learning Neural and Evolutionary Computing 41A05, 65D05, 41A25 We study the interpolation power of deep ReLU neural networks. Specifically, we consider the question of how efficiently, in terms of the number of parameters, deep ReLU networks can interpolate values at $N$ datapoints in the unit ball which are separated by a distance $δ$. We show that $Ω(N)$ parameters are required in the regime where $δ$ is exponentially small in $N$, which gives the sharp result in this regime since $O(N)$ parameters are always sufficient. This also shows that the bit-extraction technique used to prove lower bounds on the VC dimension cannot be applied to irregularly spaced datapoints. Finally, as an application we give a lower bound on the approximation rates that deep ReLU neural networks can achieve for Sobolev spaces at the embedding endpoint. |
| title | Sharp Lower Bounds on Interpolation by Deep ReLU Neural Networks at Irregularly Spaced Data |
| topic | Machine Learning Neural and Evolutionary Computing 41A05, 65D05, 41A25 |
| url | https://arxiv.org/abs/2302.00834 |