Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917881978028032 |
|---|---|
| author | Zuo, Chunsheng Guerzhoy, Pavel Guerzhoy, Michael |
| author_facet | Zuo, Chunsheng Guerzhoy, Pavel Guerzhoy, Michael |
| contents | Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00073 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings Zuo, Chunsheng Guerzhoy, Pavel Guerzhoy, Michael Computation and Language Machine Learning Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters. |
| title | Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2501.00073 |