Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918171369275392 |
|---|---|
| author | Gamba, Federica Sinha, Aman Mickus, Timothee Vazquez, Raul Bhamidipati, Patanjali Savelli, Claudio Chattopadhyay, Ahana Zanella, Laura A. Kankanampati, Yash Remesh, Binesh Arakkal Chandramania, Aryan Ashok Agarwal, Rohit Li, Chuyuan Buhnila, Ioana Mamidi, Radhika |
| author_facet | Gamba, Federica Sinha, Aman Mickus, Timothee Vazquez, Raul Bhamidipati, Patanjali Savelli, Claudio Chattopadhyay, Ahana Zanella, Laura A. Kankanampati, Yash Remesh, Binesh Arakkal Chandramania, Aryan Ashok Agarwal, Rohit Li, Chuyuan Buhnila, Ioana Mamidi, Radhika |
| contents | We introduce the CAP (Confabulations from ACL Publications) dataset, a multilingual resource for studying hallucinations in large language models (LLMs) within scientific text generation. CAP focuses on the scientific domain, where hallucinations can distort factual knowledge, as they frequently do. In this domain, however, the presence of specialized terminology, statistical reasoning, and context-dependent interpretations further exacerbates these distortions, particularly given LLMs' lack of true comprehension, limited contextual understanding, and bias toward surface-level generalization. CAP operates in a cross-lingual setting covering five high-resource languages (English, French, Hindi, Italian, and Spanish) and four low-resource languages (Bengali, Gujarati, Malayalam, and Telugu). The dataset comprises 900 curated scientific questions and over 7000 LLM-generated answers from 16 publicly available models, provided as question-answer pairs along with token sequences and corresponding logits. Each instance is annotated with a binary label indicating the presence of a scientific hallucination, denoted as a factuality error, and a fluency label, capturing issues in the linguistic quality or naturalness of the text. CAP is publicly released to facilitate advanced research on hallucination detection, multilingual evaluation of LLMs, and the development of more reliable scientific NLP systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22395 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection Gamba, Federica Sinha, Aman Mickus, Timothee Vazquez, Raul Bhamidipati, Patanjali Savelli, Claudio Chattopadhyay, Ahana Zanella, Laura A. Kankanampati, Yash Remesh, Binesh Arakkal Chandramania, Aryan Ashok Agarwal, Rohit Li, Chuyuan Buhnila, Ioana Mamidi, Radhika Computation and Language We introduce the CAP (Confabulations from ACL Publications) dataset, a multilingual resource for studying hallucinations in large language models (LLMs) within scientific text generation. CAP focuses on the scientific domain, where hallucinations can distort factual knowledge, as they frequently do. In this domain, however, the presence of specialized terminology, statistical reasoning, and context-dependent interpretations further exacerbates these distortions, particularly given LLMs' lack of true comprehension, limited contextual understanding, and bias toward surface-level generalization. CAP operates in a cross-lingual setting covering five high-resource languages (English, French, Hindi, Italian, and Spanish) and four low-resource languages (Bengali, Gujarati, Malayalam, and Telugu). The dataset comprises 900 curated scientific questions and over 7000 LLM-generated answers from 16 publicly available models, provided as question-answer pairs along with token sequences and corresponding logits. Each instance is annotated with a binary label indicating the presence of a scientific hallucination, denoted as a factuality error, and a fluency label, capturing issues in the linguistic quality or naturalness of the text. CAP is publicly released to facilitate advanced research on hallucination detection, multilingual evaluation of LLMs, and the development of more reliable scientific NLP systems. |
| title | Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detection |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.22395 |