On the Scaling Laws of Geographical Representation in Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910352463101952 |
|---|---|
| author | Godey, Nathan de la Clergerie, Éric Sagot, Benoît |
| author_facet | Godey, Nathan de la Clergerie, Éric Sagot, Benoît |
| contents | Language models have long been shown to embed geographical information in their hidden representations. This line of work has recently been revisited by extending this result to Large Language Models (LLMs). In this paper, we propose to fill the gap between well-established and recent literature by observing how geographical knowledge evolves when scaling language models. We show that geographical knowledge is observable even for tiny models, and that it scales consistently as we increase the model size. Notably, we observe that larger language models cannot mitigate the geographical bias that is inherent to the training data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_19406 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Scaling Laws of Geographical Representation in Language Models Godey, Nathan de la Clergerie, Éric Sagot, Benoît Computation and Language Artificial Intelligence Language models have long been shown to embed geographical information in their hidden representations. This line of work has recently been revisited by extending this result to Large Language Models (LLMs). In this paper, we propose to fill the gap between well-established and recent literature by observing how geographical knowledge evolves when scaling language models. We show that geographical knowledge is observable even for tiny models, and that it scales consistently as we increase the model size. Notably, we observe that larger language models cannot mitigate the geographical bias that is inherent to the training data. |
| title | On the Scaling Laws of Geographical Representation in Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.19406 |