A Theory of Feature Learning in Kernel Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908834734276608 |
|---|---|
| author | Chen, Yunlu Li, Yang Liu, Keli Ruan, Feng |
| author_facet | Chen, Yunlu Li, Yang Liu, Keli Ruan, Feng |
| contents | We study feature learning in a compositional variant of kernel ridge regression in which the predictor is applied to a learnable linear transformation of the input. When the response depends on the input only through a low-dimensional predictive subspace, we show that all global minimizers of the population objective for the linear transformation annihilate directions orthogonal to this subspace, and in certain regimes, exactly identify the subspace. Moreover, we show that global minimizers of the finite-sample objective inherit the exact same low-dimensional structure with high probability, even without any explicit penalization on the linear transformation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_11736 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | A Theory of Feature Learning in Kernel Models Chen, Yunlu Li, Yang Liu, Keli Ruan, Feng Statistics Theory Optimization and Control Machine Learning We study feature learning in a compositional variant of kernel ridge regression in which the predictor is applied to a learnable linear transformation of the input. When the response depends on the input only through a low-dimensional predictive subspace, we show that all global minimizers of the population objective for the linear transformation annihilate directions orthogonal to this subspace, and in certain regimes, exactly identify the subspace. Moreover, we show that global minimizers of the finite-sample objective inherit the exact same low-dimensional structure with high probability, even without any explicit penalization on the linear transformation. |
| title | A Theory of Feature Learning in Kernel Models |
| topic | Statistics Theory Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2310.11736 |