Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.04633 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909224428109824 |
|---|---|
| author | Du, Kevin Snæbjarnarson, Vésteinn Stoehr, Niklas White, Jennifer C. Schein, Aaron Cotterell, Ryan |
| author_facet | Du, Kevin Snæbjarnarson, Vésteinn Stoehr, Niklas White, Jennifer C. Schein, Aaron Cotterell, Ryan |
| contents | To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We hypothesize that models perform this integration in a predictable way across different questions and contexts: models will rely more on prior knowledge for questions about entities (e.g., persons, places, etc.) that they are more familiar with due to higher exposure in the training corpus, and be more easily persuaded by some contexts than others. To formalize this problem, we propose two mutual information-based metrics to measure a model's dependency on a context and on its prior about an entity: first, the persuasion score of a given context represents how much a model depends on the context in its decision, and second, the susceptibility score of a given entity represents how much the model can be swayed away from its original answer distribution about an entity. We empirically test our metrics for their validity and reliability. Finally, we explore and find a relationship between the scores and the model's expected familiarity with an entity, and provide two use cases to illustrate their benefits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04633 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Context versus Prior Knowledge in Language Models Du, Kevin Snæbjarnarson, Vésteinn Stoehr, Niklas White, Jennifer C. Schein, Aaron Cotterell, Ryan Computation and Language To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We hypothesize that models perform this integration in a predictable way across different questions and contexts: models will rely more on prior knowledge for questions about entities (e.g., persons, places, etc.) that they are more familiar with due to higher exposure in the training corpus, and be more easily persuaded by some contexts than others. To formalize this problem, we propose two mutual information-based metrics to measure a model's dependency on a context and on its prior about an entity: first, the persuasion score of a given context represents how much a model depends on the context in its decision, and second, the susceptibility score of a given entity represents how much the model can be swayed away from its original answer distribution about an entity. We empirically test our metrics for their validity and reliability. Finally, we explore and find a relationship between the scores and the model's expected familiarity with an entity, and provide two use cases to illustrate their benefits. |
| title | Context versus Prior Knowledge in Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.04633 |