Dynamics of Transient Structure in In-Context Linear Regression Transformers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916591768174592 |
|---|---|
| author | Carroll, Liam Hoogland, Jesse Farrugia-Roberts, Matthew Murfet, Daniel |
| author_facet | Carroll, Liam Hoogland, Jesse Farrugia-Roberts, Matthew Murfet, Daniel |
| contents | Modern deep neural networks display striking examples of rich internal computational structure. Uncovering principles governing the development of such structure is a priority for the science of deep learning. In this paper, we explore the transient ridge phenomenon: when transformers are trained on in-context linear regression tasks with intermediate task diversity, they initially behave like ridge regression before specializing to the tasks in their training distribution. This transition from a general solution to a specialized solution is revealed by joint trajectory principal component analysis. Further, we draw on the theory of Bayesian internal model selection to suggest a general explanation for the phenomena of transient structure in transformers, based on an evolving tradeoff between loss and complexity. We empirically validate this explanation by measuring the model complexity of our transformers as defined by the local learning coefficient. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_17745 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dynamics of Transient Structure in In-Context Linear Regression Transformers Carroll, Liam Hoogland, Jesse Farrugia-Roberts, Matthew Murfet, Daniel Machine Learning Modern deep neural networks display striking examples of rich internal computational structure. Uncovering principles governing the development of such structure is a priority for the science of deep learning. In this paper, we explore the transient ridge phenomenon: when transformers are trained on in-context linear regression tasks with intermediate task diversity, they initially behave like ridge regression before specializing to the tasks in their training distribution. This transition from a general solution to a specialized solution is revealed by joint trajectory principal component analysis. Further, we draw on the theory of Bayesian internal model selection to suggest a general explanation for the phenomena of transient structure in transformers, based on an evolving tradeoff between loss and complexity. We empirically validate this explanation by measuring the model complexity of our transformers as defined by the local learning coefficient. |
| title | Dynamics of Transient Structure in In-Context Linear Regression Transformers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2501.17745 |