FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914153728311296 |
|---|---|
| author | Marinescu, Radu Bhattacharjya, Debarun Lee, Junkyu Tchrakian, Tigran Cano, Javier Carnerero Hou, Yufang Daly, Elizabeth Pascale, Alessandra |
| author_facet | Marinescu, Radu Bhattacharjya, Debarun Lee, Junkyu Tchrakian, Tigran Cano, Javier Carnerero Hou, Yufang Daly, Elizabeth Pascale, Alessandra |
| contents | Large language models (LLMs) have achieved remarkable success in generative tasks, yet they often fall short in ensuring the factual accuracy of their outputs, thus limiting their reliability in real-world applications where correctness is critical. In this paper, we present FactReasoner, a novel neuro-symbolic based factuality assessment framework that employs probabilistic reasoning to evaluate the truthfulness of long-form generated responses. FactReasoner decomposes a response into atomic units, retrieves relevant contextual information from external knowledge sources, and models the logical relationships (e.g., entailment, contradiction) between these units and their contexts using probabilistic encodings. It then estimates the posterior probability that each atomic unit is supported by the retrieved evidence. Our experiments on both labeled and unlabeled benchmark datasets demonstrate that FactReasoner often outperforms state-of-the-art prompt-based methods in terms of factual precision and recall. Our open-source implementation is publicly available at: https://github.com/IBM/FactReasoner. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18573 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models Marinescu, Radu Bhattacharjya, Debarun Lee, Junkyu Tchrakian, Tigran Cano, Javier Carnerero Hou, Yufang Daly, Elizabeth Pascale, Alessandra Computation and Language Artificial Intelligence Large language models (LLMs) have achieved remarkable success in generative tasks, yet they often fall short in ensuring the factual accuracy of their outputs, thus limiting their reliability in real-world applications where correctness is critical. In this paper, we present FactReasoner, a novel neuro-symbolic based factuality assessment framework that employs probabilistic reasoning to evaluate the truthfulness of long-form generated responses. FactReasoner decomposes a response into atomic units, retrieves relevant contextual information from external knowledge sources, and models the logical relationships (e.g., entailment, contradiction) between these units and their contexts using probabilistic encodings. It then estimates the posterior probability that each atomic unit is supported by the retrieved evidence. Our experiments on both labeled and unlabeled benchmark datasets demonstrate that FactReasoner often outperforms state-of-the-art prompt-based methods in terms of factual precision and recall. Our open-source implementation is publicly available at: https://github.com/IBM/FactReasoner. |
| title | FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.18573 |