Transformers need glasses! Information over-squashing in language tasks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914987632492544 |
|---|---|
| author | Barbero, Federico Banino, Andrea Kapturowski, Steven Kumaran, Dharshan Araújo, João G. M. Vitvitskyi, Alex Pascanu, Razvan Veličković, Petar |
| author_facet | Barbero, Federico Banino, Andrea Kapturowski, Steven Kumaran, Dharshan Araújo, João G. M. Vitvitskyi, Alex Pascanu, Razvan Veličković, Petar |
| contents | We study how information propagates in decoder-only Transformers, which are the architectural backbone of most existing frontier large language models (LLMs). We rely on a theoretical signal propagation analysis -- specifically, we analyse the representations of the last token in the final layer of the Transformer, as this is the representation used for next-token prediction. Our analysis reveals a representational collapse phenomenon: we prove that certain distinct sequences of inputs to the Transformer can yield arbitrarily close representations in the final token. This effect is exacerbated by the low-precision floating-point formats frequently used in modern LLMs. As a result, the model is provably unable to respond to these sequences in different ways -- leading to errors in, e.g., tasks involving counting or copying. Further, we show that decoder-only Transformer language models can lose sensitivity to specific tokens in the input, which relates to the well-known phenomenon of over-squashing in graph neural networks. We provide empirical evidence supporting our claims on contemporary LLMs. Our theory also points to simple solutions towards ameliorating these issues. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04267 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Transformers need glasses! Information over-squashing in language tasks Barbero, Federico Banino, Andrea Kapturowski, Steven Kumaran, Dharshan Araújo, João G. M. Vitvitskyi, Alex Pascanu, Razvan Veličković, Petar Computation and Language Machine Learning We study how information propagates in decoder-only Transformers, which are the architectural backbone of most existing frontier large language models (LLMs). We rely on a theoretical signal propagation analysis -- specifically, we analyse the representations of the last token in the final layer of the Transformer, as this is the representation used for next-token prediction. Our analysis reveals a representational collapse phenomenon: we prove that certain distinct sequences of inputs to the Transformer can yield arbitrarily close representations in the final token. This effect is exacerbated by the low-precision floating-point formats frequently used in modern LLMs. As a result, the model is provably unable to respond to these sequences in different ways -- leading to errors in, e.g., tasks involving counting or copying. Further, we show that decoder-only Transformer language models can lose sensitivity to specific tokens in the input, which relates to the well-known phenomenon of over-squashing in graph neural networks. We provide empirical evidence supporting our claims on contemporary LLMs. Our theory also points to simple solutions towards ameliorating these issues. |
| title | Transformers need glasses! Information over-squashing in language tasks |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.04267 |