All Random Features Representations are Equivalent
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912083908493312 |
|---|---|
| author | Sernau, Luke Bonacina, Silvano Saurous, Rif A. |
| author_facet | Sernau, Luke Bonacina, Silvano Saurous, Rif A. |
| contents | Random features are a powerful technique for rewriting positive-definite kernels as linear products. They bring linear tools to bear in important nonlinear domains like KNNs and attention. Unfortunately, practical implementations require approximating an expectation, usually via sampling. This has led to the development of increasingly elaborate representations with ever lower sample error. We resolve this arms race by deriving an optimal sampling policy. Under this policy all random features representations have the same approximation error, which we show is the lowest possible. This means that we are free to choose whatever representation we please, provided we sample optimally. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_18802 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | All Random Features Representations are Equivalent Sernau, Luke Bonacina, Silvano Saurous, Rif A. Machine Learning Artificial Intelligence Random features are a powerful technique for rewriting positive-definite kernels as linear products. They bring linear tools to bear in important nonlinear domains like KNNs and attention. Unfortunately, practical implementations require approximating an expectation, usually via sampling. This has led to the development of increasingly elaborate representations with ever lower sample error. We resolve this arms race by deriving an optimal sampling policy. Under this policy all random features representations have the same approximation error, which we show is the lowest possible. This means that we are free to choose whatever representation we please, provided we sample optimally. |
| title | All Random Features Representations are Equivalent |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2406.18802 |