The Geometry of Tokens in Internal Representations of Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916571585183744 |
|---|---|
| author | Viswanathan, Karthik Gardinazzi, Yuri Panerai, Giada Cazzaniga, Alberto Biagetti, Matteo |
| author_facet | Viswanathan, Karthik Gardinazzi, Yuri Panerai, Giada Cazzaniga, Alberto Biagetti, Matteo |
| contents | We investigate the relationship between the geometry of token embeddings and their role in the next token prediction within transformer models. An important aspect of this connection uses the notion of empirical measure, which encodes the distribution of token point clouds across transformer layers and drives the evolution of token representations in the mean-field interacting picture. We use metrics such as intrinsic dimension, neighborhood overlap, and cosine similarity to observationally probe these empirical measures across layers. To validate our approach, we compare these metrics to a dataset where the tokens are shuffled, which disrupts the syntactic and semantic structure. Our findings reveal a correlation between the geometric properties of token embeddings and the cross-entropy loss of next token predictions, implying that prompts with higher loss values have tokens represented in higher-dimensional spaces. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_10573 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Geometry of Tokens in Internal Representations of Large Language Models Viswanathan, Karthik Gardinazzi, Yuri Panerai, Giada Cazzaniga, Alberto Biagetti, Matteo Computation and Language Machine Learning We investigate the relationship between the geometry of token embeddings and their role in the next token prediction within transformer models. An important aspect of this connection uses the notion of empirical measure, which encodes the distribution of token point clouds across transformer layers and drives the evolution of token representations in the mean-field interacting picture. We use metrics such as intrinsic dimension, neighborhood overlap, and cosine similarity to observationally probe these empirical measures across layers. To validate our approach, we compare these metrics to a dataset where the tokens are shuffled, which disrupts the syntactic and semantic structure. Our findings reveal a correlation between the geometric properties of token embeddings and the cross-entropy loss of next token predictions, implying that prompts with higher loss values have tokens represented in higher-dimensional spaces. |
| title | The Geometry of Tokens in Internal Representations of Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2501.10573 |