The Optimality of Kernel Classifiers in Sobolev Space
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910316201246720 |
|---|---|
| author | Lai, Jianfa Li, Zhifan Huang, Dongming Lin, Qian |
| author_facet | Lai, Jianfa Li, Zhifan Huang, Dongming Lin, Qian |
| contents | Kernel methods are widely used in machine learning, especially for classification problems. However, the theoretical analysis of kernel classification is still limited. This paper investigates the statistical performances of kernel classifiers. With some mild assumptions on the conditional probability $η(x)=\mathbb{P}(Y=1\mid X=x)$, we derive an upper bound on the classification excess risk of a kernel classifier using recent advances in the theory of kernel regression. We also obtain a minimax lower bound for Sobolev spaces, which shows the optimality of the proposed classifier. Our theoretical results can be extended to the generalization error of overparameterized neural network classifiers. To make our theoretical results more applicable in realistic settings, we also propose a simple method to estimate the interpolation smoothness of $2η(x)-1$ and apply the method to real datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01148 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Optimality of Kernel Classifiers in Sobolev Space Lai, Jianfa Li, Zhifan Huang, Dongming Lin, Qian Statistics Theory Machine Learning 62G08 (Primary), 68T07, 46E22 (secondary) G.3 Kernel methods are widely used in machine learning, especially for classification problems. However, the theoretical analysis of kernel classification is still limited. This paper investigates the statistical performances of kernel classifiers. With some mild assumptions on the conditional probability $η(x)=\mathbb{P}(Y=1\mid X=x)$, we derive an upper bound on the classification excess risk of a kernel classifier using recent advances in the theory of kernel regression. We also obtain a minimax lower bound for Sobolev spaces, which shows the optimality of the proposed classifier. Our theoretical results can be extended to the generalization error of overparameterized neural network classifiers. To make our theoretical results more applicable in realistic settings, we also propose a simple method to estimate the interpolation smoothness of $2η(x)-1$ and apply the method to real datasets. |
| title | The Optimality of Kernel Classifiers in Sobolev Space |
| topic | Statistics Theory Machine Learning 62G08 (Primary), 68T07, 46E22 (secondary) G.3 |
| url | https://arxiv.org/abs/2402.01148 |