Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913906196217856 |
|---|---|
| author | Khodja, Hichem Ammar Béchet, Frédéric Brabant, Quentin Nasr, Alexis Lecorvé, Gwénolé |
| author_facet | Khodja, Hichem Ammar Béchet, Frédéric Brabant, Quentin Nasr, Alexis Lecorvé, Gwénolé |
| contents | This paper explores the robustness of language models (LMs) to variations in the temporal context within factual knowledge. It examines whether LMs can correctly associate a temporal context with a past fact valid over a defined period, by asking them to differentiate correct from incorrect contexts. The LMs' ability to distinguish is analyzed along two dimensions: the distance of the incorrect context from the validity period and the granularity of the context. To this end, a dataset called TimeStress is introduced, enabling the evaluation of 18 diverse LMs. Results reveal that the best LM achieves a perfect distinction for only 11% of the studied facts, with errors, certainly rare, but critical that humans would not make. This work highlights the limitations of current LMs in temporal representation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01220 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations Khodja, Hichem Ammar Béchet, Frédéric Brabant, Quentin Nasr, Alexis Lecorvé, Gwénolé Computation and Language Machine Learning This paper explores the robustness of language models (LMs) to variations in the temporal context within factual knowledge. It examines whether LMs can correctly associate a temporal context with a past fact valid over a defined period, by asking them to differentiate correct from incorrect contexts. The LMs' ability to distinguish is analyzed along two dimensions: the distance of the incorrect context from the validity period and the granularity of the context. To this end, a dataset called TimeStress is introduced, enabling the evaluation of 18 diverse LMs. Results reveal that the best LM achieves a perfect distinction for only 11% of the studied facts, with errors, certainly rare, but critical that humans would not make. This work highlights the limitations of current LMs in temporal representation. |
| title | Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.01220 |