Cross-Attention and Encoder-Decoder Transformers: A Logical Characterization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917473034436608 |
|---|---|
| author | Ahvonen, Veeti Heiman, Damian Kuusisto, Antti Moreno, Miguel Selin, Matias |
| author_facet | Ahvonen, Veeti Heiman, Damian Kuusisto, Antti Moreno, Miguel Selin, Matias |
| contents | We give a novel logical characterization of encoder-decoder transformers, the foundational architecture for LLMs that also sees use in various settings that benefit from cross-attention. We study such transformers over text in the practical setting of floating-point numbers and soft-attention, characterizing them with a new temporal logic. This logic extends propositional logic with a counting global modality over the encoder input and a past modality over the decoder input. We also give an additional characterization of such transformers via a type of distributed automata, and show that our results are not limited to the specific choices in the architecture and can account for changes in, e.g., masking. Finally, we discuss encoder-decoder transformers in the autoregressive setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_07705 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Cross-Attention and Encoder-Decoder Transformers: A Logical Characterization Ahvonen, Veeti Heiman, Damian Kuusisto, Antti Moreno, Miguel Selin, Matias Logic in Computer Science Artificial Intelligence F.4.1; F.1.1; I.2.0 We give a novel logical characterization of encoder-decoder transformers, the foundational architecture for LLMs that also sees use in various settings that benefit from cross-attention. We study such transformers over text in the practical setting of floating-point numbers and soft-attention, characterizing them with a new temporal logic. This logic extends propositional logic with a counting global modality over the encoder input and a past modality over the decoder input. We also give an additional characterization of such transformers via a type of distributed automata, and show that our results are not limited to the specific choices in the architecture and can account for changes in, e.g., masking. Finally, we discuss encoder-decoder transformers in the autoregressive setting. |
| title | Cross-Attention and Encoder-Decoder Transformers: A Logical Characterization |
| topic | Logic in Computer Science Artificial Intelligence F.4.1; F.1.1; I.2.0 |
| url | https://arxiv.org/abs/2605.07705 |