Spectral Selection in Symmetric Self-Attention Dynamics
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911630506328064 |
|---|---|
| author | Kuehn, Christian Yoon, Jaeyoung |
| author_facet | Kuehn, Christian Yoon, Jaeyoung |
| contents | We study self-attention dynamics on the unit sphere as an interacting particle system arising from an idealized Transformer-type update. Under a symmetry assumption on weight matrices given by $Q^\top K=V=V^\top$, the flow admits a gradient-flow structure and an exact reformulation in the eigenbasis of $V$, revealing a spectral mode-selection mechanism. We show that the dynamics exhibits two distinct asymptotic scenarios: homogeneous alignment toward the dominant eigendirection when one positive eigenvalue strictly dominates all others in modulus, and sign-split polarization toward the most negative eigendirection when $V$ is negative definite. In particular, we obtain local stability criteria for pure-mode equilibria and global selection results in both regimes. These results provide a rigorous finite-particle description of how the spectrum of the weight matrices organizes asymptotic patterns in a symmetric self-attention flow, and highlight how the symmetric setting renders the dynamics amenable to mathematical analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_26085 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spectral Selection in Symmetric Self-Attention Dynamics Kuehn, Christian Yoon, Jaeyoung Dynamical Systems Classical Analysis and ODEs 37N99, 34D05, 34D23, 37C10, 37C75 We study self-attention dynamics on the unit sphere as an interacting particle system arising from an idealized Transformer-type update. Under a symmetry assumption on weight matrices given by $Q^\top K=V=V^\top$, the flow admits a gradient-flow structure and an exact reformulation in the eigenbasis of $V$, revealing a spectral mode-selection mechanism. We show that the dynamics exhibits two distinct asymptotic scenarios: homogeneous alignment toward the dominant eigendirection when one positive eigenvalue strictly dominates all others in modulus, and sign-split polarization toward the most negative eigendirection when $V$ is negative definite. In particular, we obtain local stability criteria for pure-mode equilibria and global selection results in both regimes. These results provide a rigorous finite-particle description of how the spectrum of the weight matrices organizes asymptotic patterns in a symmetric self-attention flow, and highlight how the symmetric setting renders the dynamics amenable to mathematical analysis. |
| title | Spectral Selection in Symmetric Self-Attention Dynamics |
| topic | Dynamical Systems Classical Analysis and ODEs 37N99, 34D05, 34D23, 37C10, 37C75 |
| url | https://arxiv.org/abs/2604.26085 |