Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2312.06528 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913375808651264 |
|---|---|
| author | Cheng, Xiang Chen, Yuxin Sra, Suvrit |
| author_facet | Cheng, Xiang Chen, Yuxin Sra, Suvrit |
| contents | Many neural network architectures are known to be Turing Complete, and can thus, in principle implement arbitrary algorithms. However, Transformers are unique in that they can implement gradient-based learning algorithms under simple parameter configurations. This paper provides theoretical and empirical evidence that (non-linear) Transformers naturally learn to implement gradient descent in function space, which in turn enable them to learn non-linear functions in context. Our results apply to a broad class of combinations of non-linear architectures and non-linear in-context learning tasks. Additionally, we show that the optimal choice of non-linear activation depends in a natural way on the class of functions that need to be learned. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_06528 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context Cheng, Xiang Chen, Yuxin Sra, Suvrit Machine Learning Many neural network architectures are known to be Turing Complete, and can thus, in principle implement arbitrary algorithms. However, Transformers are unique in that they can implement gradient-based learning algorithms under simple parameter configurations. This paper provides theoretical and empirical evidence that (non-linear) Transformers naturally learn to implement gradient descent in function space, which in turn enable them to learn non-linear functions in context. Our results apply to a broad class of combinations of non-linear architectures and non-linear in-context learning tasks. Additionally, we show that the optimal choice of non-linear activation depends in a natural way on the class of functions that need to be learned. |
| title | Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2312.06528 |