Token Homogenization under Positional Bias
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914001819009024 |
|---|---|
| author | Yusupov, Viacheslav Maksimov, Danil Alaeva, Ameliia Zaitceva, Tatiana Anna, Antipina Vasileva, Anna Liu, Chenlin Chheng, Rayuth Sazanakov, Danil Chetvergov, Andrey Ermilova, Alina Shvetsov, Egor |
| author_facet | Yusupov, Viacheslav Maksimov, Danil Alaeva, Ameliia Zaitceva, Tatiana Anna, Antipina Vasileva, Anna Liu, Chenlin Chheng, Rayuth Sazanakov, Danil Chetvergov, Andrey Ermilova, Alina Shvetsov, Egor |
| contents | This paper investigates token homogenization - the convergence of token representations toward uniformity across transformer layers and its relationship to positional bias in large language models. We empirically examine whether homogenization occurs and how positional bias amplifies this effect. Through layer-wise similarity analysis and controlled experiments, we demonstrate that tokens systematically lose distinctiveness during processing, particularly when biased toward extremal positions. Our findings confirm both the existence of homogenization and its dependence on positional attention mechanisms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_17126 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Token Homogenization under Positional Bias Yusupov, Viacheslav Maksimov, Danil Alaeva, Ameliia Zaitceva, Tatiana Anna, Antipina Vasileva, Anna Liu, Chenlin Chheng, Rayuth Sazanakov, Danil Chetvergov, Andrey Ermilova, Alina Shvetsov, Egor Computation and Language Artificial Intelligence Machine Learning This paper investigates token homogenization - the convergence of token representations toward uniformity across transformer layers and its relationship to positional bias in large language models. We empirically examine whether homogenization occurs and how positional bias amplifies this effect. Through layer-wise similarity analysis and controlled experiments, we demonstrate that tokens systematically lose distinctiveness during processing, particularly when biased toward extremal positions. Our findings confirm both the existence of homogenization and its dependence on positional attention mechanisms. |
| title | Token Homogenization under Positional Bias |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2508.17126 |