FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915421243834368 |
|---|---|
| author | Chen, Mingda Li, Yang Chen, Xilun Williams, Adina Ghosh, Gargi Yih, Scott |
| author_facet | Chen, Mingda Li, Yang Chen, Xilun Williams, Adina Ghosh, Gargi Yih, Scott |
| contents | Long-form factuality evaluation assesses the ability of models to generate accurate, comprehensive responses to short prompts. Existing benchmarks often lack human verification, leading to potential quality issues. To address this limitation, we introduce FACTORY, a large-scale, human-verified prompt set. Developed using a model-in-the-loop approach and refined by humans, FACTORY includes challenging prompts that are fact-seeking, answerable, and unambiguous. We conduct human evaluations on 6 state-of-the-art language models using FACTORY and existing datasets. Our results show that FACTORY is a challenging benchmark: approximately 40% of the claims made in the responses of SOTA models are not factual, compared to only 10% for other datasets. Our analysis identifies the strengths of FACTORY over prior benchmarks, emphasizing its reliability and the necessity for models to reason across long-tailed facts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_00109 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality Chen, Mingda Li, Yang Chen, Xilun Williams, Adina Ghosh, Gargi Yih, Scott Computation and Language Artificial Intelligence Long-form factuality evaluation assesses the ability of models to generate accurate, comprehensive responses to short prompts. Existing benchmarks often lack human verification, leading to potential quality issues. To address this limitation, we introduce FACTORY, a large-scale, human-verified prompt set. Developed using a model-in-the-loop approach and refined by humans, FACTORY includes challenging prompts that are fact-seeking, answerable, and unambiguous. We conduct human evaluations on 6 state-of-the-art language models using FACTORY and existing datasets. Our results show that FACTORY is a challenging benchmark: approximately 40% of the claims made in the responses of SOTA models are not factual, compared to only 10% for other datasets. Our analysis identifies the strengths of FACTORY over prior benchmarks, emphasizing its reliability and the necessity for models to reason across long-tailed facts. |
| title | FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.00109 |