Normalization in Attention Dynamics
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908643372302336 |
|---|---|
| author | Karagodin, Nikita Ge, Shu Polyanskiy, Yury Rigollet, Philippe |
| author_facet | Karagodin, Nikita Ge, Shu Polyanskiy, Yury Rigollet, Philippe |
| contents | We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes -- including Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT -- revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN as a particularly effective choice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22026 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Normalization in Attention Dynamics Karagodin, Nikita Ge, Shu Polyanskiy, Yury Rigollet, Philippe Machine Learning Artificial Intelligence 68T07, 35Q68, 37C10, 37N99, 82C22 We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes -- including Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT -- revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN as a particularly effective choice. |
| title | Normalization in Attention Dynamics |
| topic | Machine Learning Artificial Intelligence 68T07, 35Q68, 37C10, 37N99, 82C22 |
| url | https://arxiv.org/abs/2510.22026 |