Dimension-free deterministic equivalents and scaling laws for random feature regression
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912105123282944 |
|---|---|
| author | Defilippis, Leonardo Loureiro, Bruno Misiakiewicz, Theodor |
| author_facet | Defilippis, Leonardo Loureiro, Bruno Misiakiewicz, Theodor |
| contents | In this work we investigate the generalization performance of random feature ridge regression (RFRR). Our main contribution is a general deterministic equivalent for the test error of RFRR. Specifically, under a certain concentration property, we show that the test error is well approximated by a closed-form expression that only depends on the feature map eigenvalues. Notably, our approximation guarantee is non-asymptotic, multiplicative, and independent of the feature map dimension -- allowing for infinite-dimensional features. We expect this deterministic equivalent to hold broadly beyond our theoretical analysis, and we empirically validate its predictions on various real and synthetic datasets. As an application, we derive sharp excess error rates under standard power-law assumptions of the spectrum and target decay. In particular, we provide a tight result for the smallest number of features achieving optimal minimax error rate. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15699 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dimension-free deterministic equivalents and scaling laws for random feature regression Defilippis, Leonardo Loureiro, Bruno Misiakiewicz, Theodor Machine Learning Disordered Systems and Neural Networks In this work we investigate the generalization performance of random feature ridge regression (RFRR). Our main contribution is a general deterministic equivalent for the test error of RFRR. Specifically, under a certain concentration property, we show that the test error is well approximated by a closed-form expression that only depends on the feature map eigenvalues. Notably, our approximation guarantee is non-asymptotic, multiplicative, and independent of the feature map dimension -- allowing for infinite-dimensional features. We expect this deterministic equivalent to hold broadly beyond our theoretical analysis, and we empirically validate its predictions on various real and synthetic datasets. As an application, we derive sharp excess error rates under standard power-law assumptions of the spectrum and target decay. In particular, we provide a tight result for the smallest number of features achieving optimal minimax error rate. |
| title | Dimension-free deterministic equivalents and scaling laws for random feature regression |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2405.15699 |