The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916137327919104 |
|---|---|
| author | Razzhigaev, Anton Mikhalchuk, Matvey Goncharova, Elizaveta Oseledets, Ivan Dimitrov, Denis Kuznetsov, Andrey |
| author_facet | Razzhigaev, Anton Mikhalchuk, Matvey Goncharova, Elizaveta Oseledets, Ivan Dimitrov, Denis Kuznetsov, Andrey |
| contents | In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encoders and decoders. Our findings reveal that the anisotropy profile in transformer decoders exhibits a distinct bell-shaped curve, with the highest anisotropy concentrations in the middle layers. This pattern diverges from the more uniformly distributed anisotropy observed in encoders. In addition, we found that the intrinsic dimension of embeddings increases in the initial phases of training, indicating an expansion into higher-dimensional space. Which is then followed by a compression phase towards the end of training with dimensionality decrease, suggesting a refinement into more compact representations. Our results provide fresh insights to the understanding of encoders and decoders embedding properties. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_05928 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models Razzhigaev, Anton Mikhalchuk, Matvey Goncharova, Elizaveta Oseledets, Ivan Dimitrov, Denis Kuznetsov, Andrey Computation and Language Artificial Intelligence Information Theory Machine Learning General Topology In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encoders and decoders. Our findings reveal that the anisotropy profile in transformer decoders exhibits a distinct bell-shaped curve, with the highest anisotropy concentrations in the middle layers. This pattern diverges from the more uniformly distributed anisotropy observed in encoders. In addition, we found that the intrinsic dimension of embeddings increases in the initial phases of training, indicating an expansion into higher-dimensional space. Which is then followed by a compression phase towards the end of training with dimensionality decrease, suggesting a refinement into more compact representations. Our results provide fresh insights to the understanding of encoders and decoders embedding properties. |
| title | The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models |
| topic | Computation and Language Artificial Intelligence Information Theory Machine Learning General Topology |
| url | https://arxiv.org/abs/2311.05928 |