Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2410.18359 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908388530585600 |
|---|---|
| author | Xie, Yiqing Zhou, Wenxuan Prakash, Pradyot Jin, Di Mao, Yuning Fettes, Quintin Talebzadeh, Arya Wang, Sinong Fang, Han Rose, Carolyn Fried, Daniel Zhang, Hejia |
| author_facet | Xie, Yiqing Zhou, Wenxuan Prakash, Pradyot Jin, Di Mao, Yuning Fettes, Quintin Talebzadeh, Arya Wang, Sinong Fang, Han Rose, Carolyn Fried, Daniel Zhang, Hejia |
| contents | Factuality evaluation aims to detect factual errors produced by language models (LMs) and hence guide the development of more factual models. Towards this goal, we train a factuality evaluator, FenCE, that provides LM generators with claim-level factuality feedback. We conduct data augmentation on a combination of public judgment datasets to train FenCE to (1) generate textual critiques along with scores and (2) make claim-level judgment based on diverse source documents obtained by various tools. We then present a framework that leverages FenCE to improve the factuality of LM generators by constructing training data. Specifically, we generate a set of candidate responses, leverage FenCE to revise and score each response without introducing lesser-known facts, and train the generator by preferring highly scored revised responses. Experiments show that our data augmentation methods improve the evaluator's accuracy by 2.9% on LLM-AggreFact. With FenCE, we improve Llama2-7B-chat and Llama3-8B-chat's factuality rate by 16.86% and 14.45% on FActScore, outperforming state-of-the-art factuality finetuning methods by 8.83% and 6.96%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18359 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Model Factuality with Fine-grained Critique-based Evaluator Xie, Yiqing Zhou, Wenxuan Prakash, Pradyot Jin, Di Mao, Yuning Fettes, Quintin Talebzadeh, Arya Wang, Sinong Fang, Han Rose, Carolyn Fried, Daniel Zhang, Hejia Computation and Language Factuality evaluation aims to detect factual errors produced by language models (LMs) and hence guide the development of more factual models. Towards this goal, we train a factuality evaluator, FenCE, that provides LM generators with claim-level factuality feedback. We conduct data augmentation on a combination of public judgment datasets to train FenCE to (1) generate textual critiques along with scores and (2) make claim-level judgment based on diverse source documents obtained by various tools. We then present a framework that leverages FenCE to improve the factuality of LM generators by constructing training data. Specifically, we generate a set of candidate responses, leverage FenCE to revise and score each response without introducing lesser-known facts, and train the generator by preferring highly scored revised responses. Experiments show that our data augmentation methods improve the evaluator's accuracy by 2.9% on LLM-AggreFact. With FenCE, we improve Llama2-7B-chat and Llama3-8B-chat's factuality rate by 16.86% and 14.45% on FActScore, outperforming state-of-the-art factuality finetuning methods by 8.83% and 6.96%. |
| title | Improving Model Factuality with Fine-grained Critique-based Evaluator |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.18359 |