Toward Manifest Relationality in Transformers via Symmetry Reduction
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915811114876928 |
|---|---|
| author | François, J. Ravera, L. |
| author_facet | François, J. Ravera, L. |
| contents | Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively. While recent approaches address this by explicitly breaking symmetry, we propose a complementary framework based on symmetry reduction. We reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction. This perspective yields architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18948 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Toward Manifest Relationality in Transformers via Symmetry Reduction François, J. Ravera, L. Machine Learning Neural and Evolutionary Computing High Energy Physics - Theory Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively. While recent approaches address this by explicitly breaking symmetry, we propose a complementary framework based on symmetry reduction. We reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction. This perspective yields architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization. |
| title | Toward Manifest Relationality in Transformers via Symmetry Reduction |
| topic | Machine Learning Neural and Evolutionary Computing High Energy Physics - Theory |
| url | https://arxiv.org/abs/2602.18948 |