Quantitative Clustering in Mean-Field Transformer Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911663404351488 |
|---|---|
| author | Chen, Shi Lin, Zhengjiang Polyanskiy, Yury Rigollet, Philippe |
| author_facet | Chen, Shi Lin, Zhengjiang Polyanskiy, Yury Rigollet, Philippe |
| contents | The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this work, we investigate the long-time clustering of mean-field transformer models. More precisely, under suitable assumptions on the transformer model parameters, we establish that any suitably regular mean-field initialization synchronizes exponentially fast to a Dirac point mass, with explicit quantitative convergence rates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14697 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Quantitative Clustering in Mean-Field Transformer Models Chen, Shi Lin, Zhengjiang Polyanskiy, Yury Rigollet, Philippe Machine Learning Analysis of PDEs Dynamical Systems The evolution of tokens through deep transformer models can be modeled as an interacting particle system that has been shown to exhibit an asymptotic clustering behavior akin to the synchronization phenomenon in Kuramoto models. In this work, we investigate the long-time clustering of mean-field transformer models. More precisely, under suitable assumptions on the transformer model parameters, we establish that any suitably regular mean-field initialization synchronizes exponentially fast to a Dirac point mass, with explicit quantitative convergence rates. |
| title | Quantitative Clustering in Mean-Field Transformer Models |
| topic | Machine Learning Analysis of PDEs Dynamical Systems |
| url | https://arxiv.org/abs/2504.14697 |