Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915973987041280 |
|---|---|
| author | Feucht, Sheridan Haklay, Tal Bhalla, Usha Wurgaft, Daniel Rager, Can Sarfati, Raphaël Merullo, Jack McGrath, Thomas Lewis, Owen Lubana, Ekdeep Singh Fel, Thomas Geiger, Atticus |
| author_facet | Feucht, Sheridan Haklay, Tal Bhalla, Usha Wurgaft, Daniel Rager, Can Sarfati, Raphaël Merullo, Jack McGrath, Thomas Lewis, Owen Lubana, Ekdeep Singh Fel, Thomas Geiger, Atticus |
| contents | Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_01148 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts Feucht, Sheridan Haklay, Tal Bhalla, Usha Wurgaft, Daniel Rager, Can Sarfati, Raphaël Merullo, Jack McGrath, Thomas Lewis, Owen Lubana, Ekdeep Singh Fel, Thomas Geiger, Atticus Artificial Intelligence Computation and Language 68T50 I.2.7 Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even though Llama-3.1-8B's representations for these concepts are circularly structured, we find that instead of directly computing modular addition in the period of the cyclic concept (e.g., 12 for months), the model re-uses a generic addition mechanism across tasks that operates independently of concept-specific geometry. First, it computes the sum of its two inputs using base-10 addition (six + August=14). Then, it maps this sum back to cyclic concept space (14->February). We show that Llama-3.1-8B uses task-agnostic Fourier features to compute these sums--in fact, these features have periods that respect standard base-10 addition, e.g., 2, 5, and 10, rather than the cyclic concept period (e.g., 12 for months). Furthermore, we identify a sparse set of 28 MLP neurons re-used across all tasks (approximately 0.2% of the MLP at layer 18) that can be partitioned into disjoint clusters, each computing the sum for a Fourier feature with a different period. Our work highlights how an interplay between causal abstraction and feature geometry can deepen our mechanistic understanding of LMs. |
| title | Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts |
| topic | Artificial Intelligence Computation and Language 68T50 I.2.7 |
| url | https://arxiv.org/abs/2605.01148 |