Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908803900899328 |
|---|---|
| author | Boncoraglio, Fabrizio Erba, Vittorio Troiani, Emanuele Xu, Yizhou Krzakala, Florent Zdeborová, Lenka |
| author_facet | Boncoraglio, Fabrizio Erba, Vittorio Troiani, Emanuele Xu, Yizhou Krzakala, Florent Zdeborová, Lenka |
| contents | Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for generalization remain poorly understood. We study empirical risk minimization in a single-head tied-attention layer trained on synthetic high-dimensional sequence tasks generated from the attention-indexed model. Using tools from random matrix theory, spin-glass theory, and approximate message passing, we obtain an exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and the spectrum of the key and query matrices. Our theory predicts the full singular-value distribution of the trained query-key map, including low-rank structure and isolated spectral outliers, in qualitative agreement with observations in more realistic transformers. Finally, for targets with power-law spectra, we show that learning proceeds through sequential spectral recovery, leading to the emergence of power-law scaling laws. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24914 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws Boncoraglio, Fabrizio Erba, Vittorio Troiani, Emanuele Xu, Yizhou Krzakala, Florent Zdeborová, Lenka Machine Learning Disordered Systems and Neural Networks Information Theory Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for generalization remain poorly understood. We study empirical risk minimization in a single-head tied-attention layer trained on synthetic high-dimensional sequence tasks generated from the attention-indexed model. Using tools from random matrix theory, spin-glass theory, and approximate message passing, we obtain an exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and the spectrum of the key and query matrices. Our theory predicts the full singular-value distribution of the trained query-key map, including low-rank structure and isolated spectral outliers, in qualitative agreement with observations in more realistic transformers. Finally, for targets with power-law spectra, we show that learning proceeds through sequential spectral recovery, leading to the emergence of power-law scaling laws. |
| title | Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws |
| topic | Machine Learning Disordered Systems and Neural Networks Information Theory |
| url | https://arxiv.org/abs/2509.24914 |