Language Models Learn Universal Representations of Numbers and Here's Why You Should Care
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914496775192576 |
|---|---|
| author | Štefánik, Michal Mickus, Timothee Kadlčík, Marek Højer, Bertram Spiegel, Michal Vázquez, Raúl Sinha, Aman Kuchař, Josef Mondorf, Philipp Stenetorp, Pontus |
| author_facet | Štefánik, Michal Mickus, Timothee Kadlčík, Marek Højer, Bertram Spiegel, Michal Vázquez, Raúl Sinha, Aman Kuchař, Josef Mondorf, Philipp Stenetorp, Pontus |
| contents | Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we quantify that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26285 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Language Models Learn Universal Representations of Numbers and Here's Why You Should Care Štefánik, Michal Mickus, Timothee Kadlčík, Marek Højer, Bertram Spiegel, Michal Vázquez, Raúl Sinha, Aman Kuchař, Josef Mondorf, Philipp Stenetorp, Pontus Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations. In this work, we quantify that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different LLM families develop equivalent sinusoidal structures, and number representations are broadly interchangeable in a large swathe of experimental setups. We show that properly factoring in this characteristic is crucial when it comes to assessing how accurately LLMs encode numeric and other ordinal information, and that mechanistically enhancing this sinusoidality can also lead to reductions of LLMs' arithmetic errors. |
| title | Language Models Learn Universal Representations of Numbers and Here's Why You Should Care |
| topic | Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2510.26285 |