Relational inductive biases on attention mechanisms
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915373268336640 |
|---|---|
| author | Mijangos, Víctor Gutierrez-Vasques, Ximena Arriola, Verónica E. Rodríguez-Domínguez, Ulises Cervantes, Alexis Almanzara, José Luis |
| author_facet | Mijangos, Víctor Gutierrez-Vasques, Ximena Arriola, Verónica E. Rodríguez-Domínguez, Ulises Cervantes, Alexis Almanzara, José Luis |
| contents | Inductive learning aims to construct general models from specific examples, guided by biases that influence hypothesis selection and determine generalization capacity. In this work, we focus on characterizing the relational inductive biases present in attention mechanisms, understood as assumptions about the underlying relationships between data elements. From the perspective of geometric deep learning, we analyze the most common attention mechanisms in terms of their equivariance properties with respect to permutation subgroups, which allows us to propose a classification based on their relational biases. Under this perspective, we show that different attention layers are characterized by the underlying relationships they assume on the input data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04117 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Relational inductive biases on attention mechanisms Mijangos, Víctor Gutierrez-Vasques, Ximena Arriola, Verónica E. Rodríguez-Domínguez, Ulises Cervantes, Alexis Almanzara, José Luis Machine Learning Computation and Language Inductive learning aims to construct general models from specific examples, guided by biases that influence hypothesis selection and determine generalization capacity. In this work, we focus on characterizing the relational inductive biases present in attention mechanisms, understood as assumptions about the underlying relationships between data elements. From the perspective of geometric deep learning, we analyze the most common attention mechanisms in terms of their equivariance properties with respect to permutation subgroups, which allows us to propose a classification based on their relational biases. Under this perspective, we show that different attention layers are characterized by the underlying relationships they assume on the input data. |
| title | Relational inductive biases on attention mechanisms |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2507.04117 |