On the Anatomy of Attention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911948062326784 |
|---|---|
| author | Khatri, Nikhil Laakkonen, Tuomas Liu, Jonathon Wang-Maścianica, Vincent |
| author_facet | Khatri, Nikhil Laakkonen, Tuomas Liu, Jonathon Wang-Maścianica, Vincent |
| contents | We introduce a category-theoretic diagrammatic formalism in order to systematically relate and reason about machine learning models. Our diagrams present architectures intuitively but without loss of essential detail, where natural relationships between models are captured by graphical transformations, and important differences and similarities can be identified at a glance. In this paper, we focus on attention mechanisms: translating folklore into mathematical derivations, and constructing a taxonomy of attention variants in the literature. As a first example of an empirical investigation underpinned by our formalism, we identify recurring anatomical components of attention, which we exhaustively recombine to explore a space of variations on the attention mechanism. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02423 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Anatomy of Attention Khatri, Nikhil Laakkonen, Tuomas Liu, Jonathon Wang-Maścianica, Vincent Machine Learning Category Theory 68T01, 18M30 I.2.6 We introduce a category-theoretic diagrammatic formalism in order to systematically relate and reason about machine learning models. Our diagrams present architectures intuitively but without loss of essential detail, where natural relationships between models are captured by graphical transformations, and important differences and similarities can be identified at a glance. In this paper, we focus on attention mechanisms: translating folklore into mathematical derivations, and constructing a taxonomy of attention variants in the literature. As a first example of an empirical investigation underpinned by our formalism, we identify recurring anatomical components of attention, which we exhaustively recombine to explore a space of variations on the attention mechanism. |
| title | On the Anatomy of Attention |
| topic | Machine Learning Category Theory 68T01, 18M30 I.2.6 |
| url | https://arxiv.org/abs/2407.02423 |