Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911662667202560 |
|---|---|
| author | Isobe, Noboru Inoue, Daisuke Imaizumi, Masaaki |
| author_facet | Isobe, Noboru Inoue, Daisuke Imaizumi, Masaaki |
| contents | Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing mean-field analyses largely treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under $L^2$ regularization. We find and analyze a training-induced phase in the dynamics: after initially following attention-driven clustering, the token distribution can leave the clustered regime near the final layers. Our mathematical analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention. More broadly, our results point toward a training-aware mean-field theory of Transformer dynamics, in which training and inference dynamics are treated together. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_07772 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers Isobe, Noboru Inoue, Daisuke Imaizumi, Masaaki Machine Learning Analysis of PDEs Dynamical Systems Optimization and Control 49N80, 35Q84, 49Q22, 68T07 Transformers perform inference by iteratively transforming token representations across layers. This layerwise computation has been studied empirically, and recent mean-field theories of Transformer dynamics explain how attention can drive token distributions toward clustering. However, existing mean-field analyses largely treat model parameters as prescribed, leaving open how training reshapes this clustering picture. We study this question in a noisy mean-field Transformer in which only a parameter-linear FFN is trained under $L^2$ regularization. We find and analyze a training-induced phase in the dynamics: after initially following attention-driven clustering, the token distribution can leave the clustered regime near the final layers. Our mathematical analysis is based on an entropy-regularized interaction energy that captures the clustering bias of attention. More broadly, our results point toward a training-aware mean-field theory of Transformer dynamics, in which training and inference dynamics are treated together. |
| title | Training-Induced Escape from Token Clustering in a Mean-Field Formulation of Transformers |
| topic | Machine Learning Analysis of PDEs Dynamical Systems Optimization and Control 49N80, 35Q84, 49Q22, 68T07 |
| url | https://arxiv.org/abs/2605.07772 |