Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910463636275200 |
|---|---|
| author | Cheng, Tin Sum Lucchi, Aurelien Kratsios, Anastasis Belius, David |
| author_facet | Cheng, Tin Sum Lucchi, Aurelien Kratsios, Anastasis Belius, David |
| contents | We derive new bounds for the condition number of kernel matrices, which we then use to enhance existing non-asymptotic test error bounds for kernel ridgeless regression (KRR) in the over-parameterized regime for a fixed input dimension. For kernels with polynomial spectral decay, we recover the bound from previous work; for exponential decay, our bound is non-trivial and novel. Our contribution is two-fold: (i) we rigorously prove the phenomena of tempered overfitting and catastrophic overfitting under the sub-Gaussian design assumption, closing an existing gap in the literature; (ii) we identify that the independence of the features plays an important role in guaranteeing tempered overfitting, raising concerns about approximating KRR generalization using the Gaussian design assumption in previous literature. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01297 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum Cheng, Tin Sum Lucchi, Aurelien Kratsios, Anastasis Belius, David Machine Learning We derive new bounds for the condition number of kernel matrices, which we then use to enhance existing non-asymptotic test error bounds for kernel ridgeless regression (KRR) in the over-parameterized regime for a fixed input dimension. For kernels with polynomial spectral decay, we recover the bound from previous work; for exponential decay, our bound is non-trivial and novel. Our contribution is two-fold: (i) we rigorously prove the phenomena of tempered overfitting and catastrophic overfitting under the sub-Gaussian design assumption, closing an existing gap in the literature; (ii) we identify that the independence of the features plays an important role in guaranteeing tempered overfitting, raising concerns about approximating KRR generalization using the Gaussian design assumption in previous literature. |
| title | Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2402.01297 |