Average Attention Transformers and Arithmetic Circuits
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915983791226880 |
|---|---|
| author | Ehrmuth, Lena Strieker, Laura |
| author_facet | Ehrmuth, Lena Strieker, Laura |
| contents | We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circuits if they are given as an input to an encoder. The circuit families that can be simulated this way have constant depth while using unbounded addition, binary multiplication and sign gates. The transformers we use have arithmetic circuits instead of feed-forward networks. With typical average attention the functions they compute are also computed by the same class of circuit families. Our results hold for transformers over the reals, rationals and any ring in between the two. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04683 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Average Attention Transformers and Arithmetic Circuits Ehrmuth, Lena Strieker, Laura Computational Complexity Artificial Intelligence Machine Learning F.1.1; F.1.3; I.2.m We analyse the computational power of transformer encoders as sequence-to-sequence functions on vectors. We show that average hard attention can be used to simulate arithmetic circuits if they are given as an input to an encoder. The circuit families that can be simulated this way have constant depth while using unbounded addition, binary multiplication and sign gates. The transformers we use have arithmetic circuits instead of feed-forward networks. With typical average attention the functions they compute are also computed by the same class of circuit families. Our results hold for transformers over the reals, rationals and any ring in between the two. |
| title | Average Attention Transformers and Arithmetic Circuits |
| topic | Computational Complexity Artificial Intelligence Machine Learning F.1.1; F.1.3; I.2.m |
| url | https://arxiv.org/abs/2605.04683 |