The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913683737673728 |
|---|---|
| author | El-Shangiti, Ahmed Oumar Hiraoka, Tatsuya AlQuabeh, Hilal Heinzerling, Benjamin Inui, Kentaro |
| author_facet | El-Shangiti, Ahmed Oumar Hiraoka, Tatsuya AlQuabeh, Hilal Heinzerling, Benjamin Inui, Kentaro |
| contents | This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of the embedding space when answering questions involving numeric comparisons, e.g., Was Cristiano born before Messi? We first identified, using partial least squares regression, these subspaces, which effectively encode the numerical attributes associated with the entities in comparison prompts. Further, we demonstrate causality, by intervening in these subspaces to manipulate hidden states, thereby altering the LLM's comparison outcomes. Experiments conducted on three different LLMs showed that our results hold across different numerical attributes, indicating that LLMs utilize the linearly encoded information for numerical reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13194 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces El-Shangiti, Ahmed Oumar Hiraoka, Tatsuya AlQuabeh, Hilal Heinzerling, Benjamin Inui, Kentaro Computation and Language This paper investigates whether large language models (LLMs) utilize numerical attributes encoded in a low-dimensional subspace of the embedding space when answering questions involving numeric comparisons, e.g., Was Cristiano born before Messi? We first identified, using partial least squares regression, these subspaces, which effectively encode the numerical attributes associated with the entities in comparison prompts. Further, we demonstrate causality, by intervening in these subspaces to manipulate hidden states, thereby altering the LLM's comparison outcomes. Experiments conducted on three different LLMs showed that our results hold across different numerical attributes, indicating that LLMs utilize the linearly encoded information for numerical reasoning. |
| title | The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.13194 |