Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.06256 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909826398814208 |
|---|---|
| author | Bratulić, Jelena Mittal, Sudhanshu Hoffmann, David T. Böhm, Samuel Schirrmeister, Robin Tibor Ball, Tonio Rupprecht, Christian Brox, Thomas |
| author_facet | Bratulić, Jelena Mittal, Sudhanshu Hoffmann, David T. Böhm, Samuel Schirrmeister, Robin Tibor Ball, Tonio Rupprecht, Christian Brox, Thomas |
| contents | Large Language Models (LLMs) exhibit In-Context Learning (ICL), which enables the model to perform new tasks conditioning only on the examples provided in the context without updating the model's weights. While ICL offers fast adaptation across natural language tasks and domains, its emergence is less straightforward for modalities beyond text. In this work, we systematically uncover properties present in LLMs that support the emergence of ICL for autoregressive models and various modalities by promoting the learning of the needed mechanisms for ICL. We identify exact token repetitions in the training data sequences as an important factor for ICL. Such repetitions further improve stability and reduce transiency in ICL performance. Moreover, we emphasise the significance of training task difficulty for the emergence of ICL. Finally, by applying our novel insights on ICL emergence, we unlock ICL capabilities for various visual datasets and a more challenging EEG classification task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06256 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling Bratulić, Jelena Mittal, Sudhanshu Hoffmann, David T. Böhm, Samuel Schirrmeister, Robin Tibor Ball, Tonio Rupprecht, Christian Brox, Thomas Computation and Language Artificial Intelligence Machine Learning Large Language Models (LLMs) exhibit In-Context Learning (ICL), which enables the model to perform new tasks conditioning only on the examples provided in the context without updating the model's weights. While ICL offers fast adaptation across natural language tasks and domains, its emergence is less straightforward for modalities beyond text. In this work, we systematically uncover properties present in LLMs that support the emergence of ICL for autoregressive models and various modalities by promoting the learning of the needed mechanisms for ICL. We identify exact token repetitions in the training data sequences as an important factor for ICL. Such repetitions further improve stability and reduce transiency in ICL performance. Moreover, we emphasise the significance of training task difficulty for the emergence of ICL. Finally, by applying our novel insights on ICL emergence, we unlock ICL capabilities for various visual datasets and a more challenging EEG classification task. |
| title | Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2501.06256 |