A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913119911018496 |
|---|---|
| author | Hayase, Tomohiro Karakida, Ryo |
| author_facet | Hayase, Tomohiro Karakida, Ryo |
| contents | Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest conflicting inverse-temperature laws for the context length $n$, ranging from $(\log n)^{1/2}$ to $\log n$ and $(\log n)^2$. We provide a general theory showing that the desirable scale is determined by the gap-counting function $N_n$ of each attention row. Counting how many competitors lie within each gap from the maximum, we define an upper-tail accumulation scale and prove that it gives the critical inverse-temperature scale for softmax concentration: below this scale, the top competitors remain unseparated, whereas above it, the attention entropy collapses. This framework unifies prior scaling laws as different $N_n$ and yields a direct diagnostic for attention-score families, from idealized theoretical models to more practical transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12697 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention Hayase, Tomohiro Karakida, Ryo Machine Learning Probability 68T07, 41A60, 82B26, 60G70 G.3; I.2.7 Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest conflicting inverse-temperature laws for the context length $n$, ranging from $(\log n)^{1/2}$ to $\log n$ and $(\log n)^2$. We provide a general theory showing that the desirable scale is determined by the gap-counting function $N_n$ of each attention row. Counting how many competitors lie within each gap from the maximum, we define an upper-tail accumulation scale and prove that it gives the critical inverse-temperature scale for softmax concentration: below this scale, the top competitors remain unseparated, whereas above it, the attention entropy collapses. This framework unifies prior scaling laws as different $N_n$ and yields a direct diagnostic for attention-score families, from idealized theoretical models to more practical transformers. |
| title | A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention |
| topic | Machine Learning Probability 68T07, 41A60, 82B26, 60G70 G.3; I.2.7 |
| url | https://arxiv.org/abs/2605.12697 |