Learning without training: The implicit dynamics of in-context learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909973770928128 |
|---|---|
| author | Dherin, Benoit Munn, Michael Mazzawi, Hanna Wunder, Michael Gonzalvo, Javier |
| author_facet | Dherin, Benoit Munn, Michael Mazzawi, Hanna Wunder, Michael Gonzalvo, Javier |
| contents | One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP, allows the transformer block to implicitly modify the weights of the MLP layer according to the context. We argue through theory and experimentation that this simple mechanism may be the reason why LLMs can learn in-context and not only during training. Specifically, we show how a transformer block implicitly transforms a context into a low-rank weight-update of its MLP layer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_16003 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning without training: The implicit dynamics of in-context learning Dherin, Benoit Munn, Michael Mazzawi, Hanna Wunder, Michael Gonzalvo, Javier Computation and Language Machine Learning One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP, allows the transformer block to implicitly modify the weights of the MLP layer according to the context. We argue through theory and experimentation that this simple mechanism may be the reason why LLMs can learn in-context and not only during training. Specifically, we show how a transformer block implicitly transforms a context into a low-rank weight-update of its MLP layer. |
| title | Learning without training: The implicit dynamics of in-context learning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.16003 |