UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910393972031488 |
|---|---|
| author | Yang, Yijun He, Jie Chen, Pinzhen Gutiérrez-Basulto, Víctor Pan, Jeff Z. |
| author_facet | Yang, Yijun He, Jie Chen, Pinzhen Gutiérrez-Basulto, Víctor Pan, Jeff Z. |
| contents | Several recent papers have investigated the potential of language models as knowledge bases as well as the existence of severe biases when extracting factual knowledge. In this work, we focus on the factual probing performance over unseen prompts from tuning, and using a probabilistic view we show the inherent misalignment between pre-training and downstream tuning objectives in language models for probing knowledge. We hypothesize that simultaneously debiasing these objectives can be the key to generalisation over unseen prompts. We propose an adapter-based framework, UniArk, for generalised and consistent factual knowledge extraction through simple methods without introducing extra parameters. Extensive experiments show that UniArk can significantly improve the model's out-of-domain generalisation as well as consistency under various prompts. Additionally, we construct ParaTrex, a large-scale and diverse dataset for measuring the inconsistency and out-of-domain generation of models. Further, ParaTrex offers a reference method for constructing paraphrased datasets using large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01253 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing Yang, Yijun He, Jie Chen, Pinzhen Gutiérrez-Basulto, Víctor Pan, Jeff Z. Computation and Language Several recent papers have investigated the potential of language models as knowledge bases as well as the existence of severe biases when extracting factual knowledge. In this work, we focus on the factual probing performance over unseen prompts from tuning, and using a probabilistic view we show the inherent misalignment between pre-training and downstream tuning objectives in language models for probing knowledge. We hypothesize that simultaneously debiasing these objectives can be the key to generalisation over unseen prompts. We propose an adapter-based framework, UniArk, for generalised and consistent factual knowledge extraction through simple methods without introducing extra parameters. Extensive experiments show that UniArk can significantly improve the model's out-of-domain generalisation as well as consistency under various prompts. Additionally, we construct ParaTrex, a large-scale and diverse dataset for measuring the inconsistency and out-of-domain generation of models. Further, ParaTrex offers a reference method for constructing paraphrased datasets using large language models. |
| title | UniArk: Improving Generalisation and Consistency for Factual Knowledge Extraction through Debiasing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.01253 |