MuLan: A Study of Fact Mutability in Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909160380039168 |
|---|---|
| author | Fierro, Constanza Garneau, Nicolas Bugliarello, Emanuele Kementchedjhieva, Yova Søgaard, Anders |
| author_facet | Fierro, Constanza Garneau, Nicolas Bugliarello, Emanuele Kementchedjhieva, Yova Søgaard, Anders |
| contents | Facts are subject to contingencies and can be true or false in different circumstances. One such contingency is time, wherein some facts mutate over a given period, e.g., the president of a country or the winner of a championship. Trustworthy language models ideally identify mutable facts as such and process them accordingly. We create MuLan, a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. We hypothesize that mutable facts are encoded differently than immutable ones, hence being easier to update. In a detailed evaluation of six popular large language models, we consistently find differences in the LLMs' confidence, representations, and update behavior, depending on the mutability of a fact. Our findings should inform future work on the injection of and induction of time-contingent knowledge to/from LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03036 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MuLan: A Study of Fact Mutability in Language Models Fierro, Constanza Garneau, Nicolas Bugliarello, Emanuele Kementchedjhieva, Yova Søgaard, Anders Computation and Language Facts are subject to contingencies and can be true or false in different circumstances. One such contingency is time, wherein some facts mutate over a given period, e.g., the president of a country or the winner of a championship. Trustworthy language models ideally identify mutable facts as such and process them accordingly. We create MuLan, a benchmark for evaluating the ability of English language models to anticipate time-contingency, covering both 1:1 and 1:N relations. We hypothesize that mutable facts are encoded differently than immutable ones, hence being easier to update. In a detailed evaluation of six popular large language models, we consistently find differences in the LLMs' confidence, representations, and update behavior, depending on the mutability of a fact. Our findings should inform future work on the injection of and induction of time-contingent knowledge to/from LLMs. |
| title | MuLan: A Study of Fact Mutability in Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.03036 |