Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912667185184768 |
|---|---|
| author | Kadlčík, Marek Štefánik, Michal Mickus, Timothee Spiegel, Michal Kuchař, Josef |
| author_facet | Kadlčík, Marek Štefánik, Michal Mickus, Timothee Spiegel, Michal Kuchař, Josef |
| contents | Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models' representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. However, we observe that previous probing methods are inadequate for the emergent structure of learned number embeddings with sinusoidal patterns.
In response, we propose a novel probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. This proves that after the sole pre-training, LMs represent numbers with remarkable precision. Finally, we find that the embeddings' precision, judged by our probe's accuracy, explains a large portion of LM's errors in elementary arithmetic, and show that aligning the embeddings with the pattern our probes discover can mitigate these errors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08966 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers Kadlčík, Marek Štefánik, Michal Mickus, Timothee Spiegel, Michal Kuchař, Josef Computation and Language Machine Learning Neural and Evolutionary Computing Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models' representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. However, we observe that previous probing methods are inadequate for the emergent structure of learned number embeddings with sinusoidal patterns. In response, we propose a novel probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. This proves that after the sole pre-training, LMs represent numbers with remarkable precision. Finally, we find that the embeddings' precision, judged by our probe's accuracy, explains a large portion of LM's errors in elementary arithmetic, and show that aligning the embeddings with the pattern our probes discover can mitigate these errors. |
| title | Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers |
| topic | Computation and Language Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2506.08966 |