Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.04480 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912809681420288 |
|---|---|
| author | Gurnee, Wes Ameisen, Emmanuel Kauvar, Isaac Tarng, Julius Pearce, Adam Olah, Chris Batson, Joshua |
| author_facet | Gurnee, Wes Ameisen, Emmanuel Kauvar, Isaac Tarng, Julius Pearce, Adam Olah, Chris Batson, Joshua |
| contents | Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04480 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Models Manipulate Manifolds: The Geometry of a Counting Task Gurnee, Wes Ameisen, Emmanuel Kauvar, Isaac Tarng, Julius Pearce, Adam Olah, Chris Batson, Joshua Machine Learning Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability. |
| title | When Models Manipulate Manifolds: The Geometry of a Counting Task |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.04480 |