Learning to (Learn at Test Time)
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909063919435776 |
|---|---|
| author | Sun, Yu Li, Xinhao Dalal, Karan Hsu, Chloe Koyejo, Sanmi Guestrin, Carlos Wang, Xiaolong Hashimoto, Tatsunori Chen, Xinlei |
| author_facet | Sun, Yu Li, Xinhao Dalal, Karan Hsu, Chloe Koyejo, Sanmi Guestrin, Carlos Wang, Xiaolong Hashimoto, Tatsunori Chen, Xinlei |
| contents | We reformulate the problem of supervised learning as learning to learn with two nested loops (i.e. learning problems). The inner loop learns on each individual instance with self-supervision before final prediction. The outer loop learns the self-supervised task used by the inner loop, such that its final prediction improves. Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator. For practical comparison with linear or self-attention layers, we replace each of them in a transformer with an inner loop, so our outer loop is equivalent to training the architecture. When each inner-loop learner is a neural network, our approach vastly outperforms transformers with linear attention on ImageNet from 224 x 224 raw pixels in both accuracy and FLOPs, while (regular) transformers cannot run. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_13807 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Learning to (Learn at Test Time) Sun, Yu Li, Xinhao Dalal, Karan Hsu, Chloe Koyejo, Sanmi Guestrin, Carlos Wang, Xiaolong Hashimoto, Tatsunori Chen, Xinlei Machine Learning We reformulate the problem of supervised learning as learning to learn with two nested loops (i.e. learning problems). The inner loop learns on each individual instance with self-supervision before final prediction. The outer loop learns the self-supervised task used by the inner loop, such that its final prediction improves. Our inner loop turns out to be equivalent to linear attention when the inner-loop learner is only a linear model, and to self-attention when it is a kernel estimator. For practical comparison with linear or self-attention layers, we replace each of them in a transformer with an inner loop, so our outer loop is equivalent to training the architecture. When each inner-loop learner is a neural network, our approach vastly outperforms transformers with linear attention on ImageNet from 224 x 224 raw pixels in both accuracy and FLOPs, while (regular) transformers cannot run. |
| title | Learning to (Learn at Test Time) |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2310.13807 |