Beyond Position: the emergence of wavelet-like properties in Transformers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908393853157376 |
|---|---|
| author | Ruscio, Valeria Nanni, Umberto Silvestri, Fabrizio |
| author_facet | Ruscio, Valeria Nanni, Umberto Silvestri, Fabrizio |
| contents | This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical limitations. Through an analysis spanning model scales, architectures, and training checkpoints, we show that attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. We demonstrate that this scale-invariant behavior is unique to RoPE, emerges through distinct evolutionary phases during training, and statistically adheres to the fundamental uncertainty principle. Our findings suggest that the effectiveness of modern Transformers stems from their remarkable ability to spontaneously develop optimal, multi-resolution decompositions to address inherent architectural constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18067 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Position: the emergence of wavelet-like properties in Transformers Ruscio, Valeria Nanni, Umberto Silvestri, Fabrizio Machine Learning Artificial Intelligence This paper studies how Transformer models with Rotary Position Embeddings (RoPE) develop emergent, wavelet-like properties that compensate for the positional encoding's theoretical limitations. Through an analysis spanning model scales, architectures, and training checkpoints, we show that attention heads evolve to implement multi-resolution processing analogous to wavelet transforms. We demonstrate that this scale-invariant behavior is unique to RoPE, emerges through distinct evolutionary phases during training, and statistically adheres to the fundamental uncertainty principle. Our findings suggest that the effectiveness of modern Transformers stems from their remarkable ability to spontaneously develop optimal, multi-resolution decompositions to address inherent architectural constraints. |
| title | Beyond Position: the emergence of wavelet-like properties in Transformers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.18067 |