The Geometry of Thought: Disclosing the Transformer as a Tropical Polynomial Circuit
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908766611439616 |
|---|---|
| author | Alpay, Faruk Senturk, Bilge |
| author_facet | Alpay, Faruk Senturk, Bilge |
| contents | We prove that the Transformer self-attention mechanism in the high-confidence regime ($β\to \infty$, where $β$ is an inverse temperature) operates in the tropical semiring (max-plus algebra). In particular, we show that taking the tropical limit of the softmax attention converts it into a tropical matrix product. This reveals that the Transformer's forward pass is effectively executing a dynamic programming recurrence (specifically, a Bellman-Ford path-finding update) on a latent graph defined by token similarities. Our theoretical result provides a new geometric perspective for chain-of-thought reasoning: it emerges from an inherent shortest-path (or longest-path) algorithm being carried out within the network's computation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09775 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Geometry of Thought: Disclosing the Transformer as a Tropical Polynomial Circuit Alpay, Faruk Senturk, Bilge Machine Learning Computation and Language 68T07, 14T05, 90C35, 90C39, 05C85 I.2.6; I.2.7; I.2.8; G.2.2; F.2.2; F.1.1 We prove that the Transformer self-attention mechanism in the high-confidence regime ($β\to \infty$, where $β$ is an inverse temperature) operates in the tropical semiring (max-plus algebra). In particular, we show that taking the tropical limit of the softmax attention converts it into a tropical matrix product. This reveals that the Transformer's forward pass is effectively executing a dynamic programming recurrence (specifically, a Bellman-Ford path-finding update) on a latent graph defined by token similarities. Our theoretical result provides a new geometric perspective for chain-of-thought reasoning: it emerges from an inherent shortest-path (or longest-path) algorithm being carried out within the network's computation. |
| title | The Geometry of Thought: Disclosing the Transformer as a Tropical Polynomial Circuit |
| topic | Machine Learning Computation and Language 68T07, 14T05, 90C35, 90C39, 05C85 I.2.6; I.2.7; I.2.8; G.2.2; F.2.2; F.1.1 |
| url | https://arxiv.org/abs/2601.09775 |