Uncovering mesa-optimization algorithms in Transformers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917802740285440 |
|---|---|
| author | von Oswald, Johannes Schlegel, Maximilian Meulemans, Alexander Kobayashi, Seijin Niklasson, Eyvind Zucchet, Nicolas Scherrer, Nino Miller, Nolan Sandler, Mark Arcas, Blaise Agüera y Vladymyrov, Max Pascanu, Razvan Sacramento, João |
| author_facet | von Oswald, Johannes Schlegel, Maximilian Meulemans, Alexander Kobayashi, Seijin Niklasson, Eyvind Zucchet, Nicolas Scherrer, Nino Miller, Nolan Sandler, Mark Arcas, Blaise Agüera y Vladymyrov, Max Pascanu, Razvan Sacramento, João |
| contents | Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so. The origins of this phenomenon are still poorly understood. Here we analyze a series of Transformer models trained to perform synthetic sequence prediction tasks, and discover that standard next-token prediction error minimization gives rise to a subsidiary learning algorithm that adjusts the model as new inputs are revealed. We show that this process corresponds to gradient-based optimization of a principled objective function, which leads to strong generalization performance on unseen sequences. Our findings explain in-context learning as a product of autoregressive loss minimization and inform the design of new optimization-based Transformer layers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_05858 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Uncovering mesa-optimization algorithms in Transformers von Oswald, Johannes Schlegel, Maximilian Meulemans, Alexander Kobayashi, Seijin Niklasson, Eyvind Zucchet, Nicolas Scherrer, Nino Miller, Nolan Sandler, Mark Arcas, Blaise Agüera y Vladymyrov, Max Pascanu, Razvan Sacramento, João Machine Learning Artificial Intelligence Some autoregressive models exhibit in-context learning capabilities: being able to learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so. The origins of this phenomenon are still poorly understood. Here we analyze a series of Transformer models trained to perform synthetic sequence prediction tasks, and discover that standard next-token prediction error minimization gives rise to a subsidiary learning algorithm that adjusts the model as new inputs are revealed. We show that this process corresponds to gradient-based optimization of a principled objective function, which leads to strong generalization performance on unseen sequences. Our findings explain in-context learning as a product of autoregressive loss minimization and inform the design of new optimization-based Transformer layers. |
| title | Uncovering mesa-optimization algorithms in Transformers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2309.05858 |