Kinetic theory for Transformers and the lost-in-the-middle phenomenon
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914548589527040 |
|---|---|
| author | Duerinckx, Mitia Geshkovski, Borjan Rossi, Stefano |
| author_facet | Duerinckx, Mitia Geshkovski, Borjan Rossi, Stefano |
| contents | We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed \emph{lost-in-the-middle} phenomenon: the token retrieval profile, as a function of the source position in the prompt, is $\mathsf{U}$-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09213 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Kinetic theory for Transformers and the lost-in-the-middle phenomenon Duerinckx, Mitia Geshkovski, Borjan Rossi, Stefano Analysis of PDEs Machine Learning Probability We study causal self-attention dynamics -- a toy model for decoder Transformers -- which we interpret as a non-exchangeable interacting particle system. Adapting cumulant expansions to the triangular causal dependency structure of the model, and appealing to non-hierarchical methods to estimate correlations using Glauber calculus, we prove a quantitative mean-field limit result and a next-order characterization of correlations. For iid uniformly distributed tokens, the limiting correlation equation can be solved in closed form and we obtain a rigorous explanation of the empirically observed \emph{lost-in-the-middle} phenomenon: the token retrieval profile, as a function of the source position in the prompt, is $\mathsf{U}$-shaped, with primacy, recency, and a unique interior minimum under an explicit smallness condition. |
| title | Kinetic theory for Transformers and the lost-in-the-middle phenomenon |
| topic | Analysis of PDEs Machine Learning Probability |
| url | https://arxiv.org/abs/2605.09213 |