Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912149615411200 |
|---|---|
| author | Zhang, Zixuan Zhang, Kaiqi Chen, Minshuo Takeda, Yuma Wang, Mengdi Zhao, Tuo Wang, Yu-Xiang |
| author_facet | Zhang, Zixuan Zhang, Kaiqi Chen, Minshuo Takeda, Yuma Wang, Mengdi Zhao, Tuo Wang, Yu-Xiang |
| contents | Convolutional residual neural networks (ConvResNets), though overparameterized, can achieve remarkable prediction performance in practice, which cannot be well explained by conventional wisdom. To bridge this gap, we study the performance of ConvResNeXts, which cover ConvResNets as a special case, trained with weight decay from the perspective of nonparametric classification. Our analysis allows for infinitely many building blocks in ConvResNeXts, and shows that weight decay implicitly enforces sparsity on these blocks. Specifically, we consider a smooth target function supported on a low-dimensional manifold, then prove that ConvResNeXts can adapt to the function smoothness and low-dimensional structures and efficiently learn the function without suffering from the curse of dimensionality. Our findings partially justify the advantage of overparameterized ConvResNeXts over conventional machine learning models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_01649 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks Zhang, Zixuan Zhang, Kaiqi Chen, Minshuo Takeda, Yuma Wang, Mengdi Zhao, Tuo Wang, Yu-Xiang Machine Learning Convolutional residual neural networks (ConvResNets), though overparameterized, can achieve remarkable prediction performance in practice, which cannot be well explained by conventional wisdom. To bridge this gap, we study the performance of ConvResNeXts, which cover ConvResNets as a special case, trained with weight decay from the perspective of nonparametric classification. Our analysis allows for infinitely many building blocks in ConvResNeXts, and shows that weight decay implicitly enforces sparsity on these blocks. Specifically, we consider a smooth target function supported on a low-dimensional manifold, then prove that ConvResNeXts can adapt to the function smoothness and low-dimensional structures and efficiently learn the function without suffering from the curse of dimensionality. Our findings partially justify the advantage of overparameterized ConvResNeXts over conventional machine learning models. |
| title | Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2307.01649 |