C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917985127497728 |
|---|---|
| author | Zhang, Xu Liu, Zhifei Wang, Jiahao Zhang, Huixuan Xu, Fan Zhang, Junzhe Wan, Xiaojun |
| author_facet | Zhang, Xu Liu, Zhifei Wang, Jiahao Zhang, Huixuan Xu, Fan Zhang, Junzhe Wan, Xiaojun |
| contents | Despite the rapid advancement of large language models, they remain highly susceptible to generating hallucinations, which significantly hinders their widespread application. Hallucination research requires dynamic and fine-grained evaluation. However, most existing hallucination benchmarks (especially in Chinese language) rely on human annotations, making automatical and cost-effective hallucination evaluation challenging. To address this, we introduce HaluAgent, an agentic framework that automatically constructs fine-grained QA dataset based on some knowledge documents. Our experiments demonstrate that the manually designed rules and prompt optimization can improve the quality of generated data. Using HaluAgent, we construct C-FAITH, a Chinese QA hallucination benchmark created from 1,399 knowledge documents obtained from web scraping, totaling 60,702 entries. We comprehensively evaluate 16 mainstream LLMs with our proposed C-FAITH, providing detailed experimental results and analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_10167 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluation Zhang, Xu Liu, Zhifei Wang, Jiahao Zhang, Huixuan Xu, Fan Zhang, Junzhe Wan, Xiaojun Computation and Language Artificial Intelligence Despite the rapid advancement of large language models, they remain highly susceptible to generating hallucinations, which significantly hinders their widespread application. Hallucination research requires dynamic and fine-grained evaluation. However, most existing hallucination benchmarks (especially in Chinese language) rely on human annotations, making automatical and cost-effective hallucination evaluation challenging. To address this, we introduce HaluAgent, an agentic framework that automatically constructs fine-grained QA dataset based on some knowledge documents. Our experiments demonstrate that the manually designed rules and prompt optimization can improve the quality of generated data. Using HaluAgent, we construct C-FAITH, a Chinese QA hallucination benchmark created from 1,399 knowledge documents obtained from web scraping, totaling 60,702 entries. We comprehensively evaluate 16 mainstream LLMs with our proposed C-FAITH, providing detailed experimental results and analysis. |
| title | C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.10167 |