ArxEval: Evaluating Retrieval and Generation in Language Models for Scientific Literature
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916576970670080 |
|---|---|
| author | Sinha, Aarush Virk, Viraj Chakraborty, Dipshikha Sreeja, P. S. |
| author_facet | Sinha, Aarush Virk, Viraj Chakraborty, Dipshikha Sreeja, P. S. |
| contents | Language Models [LMs] are now playing an increasingly large role in information generation and synthesis; the representation of scientific knowledge in these systems needs to be highly accurate. A prime challenge is hallucination; that is, generating apparently plausible but actually false information, including invented citations and nonexistent research papers. This kind of inaccuracy is dangerous in all the domains that require high levels of factual correctness, such as academia and education. This work presents a pipeline for evaluating the frequency with which language models hallucinate in generating responses in the scientific literature. We propose ArxEval, an evaluation pipeline with two tasks using ArXiv as a repository: Jumbled Titles and Mixed Titles. Our evaluation includes fifteen widely used language models and provides comparative insights into their reliability in handling scientific literature. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_10483 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ArxEval: Evaluating Retrieval and Generation in Language Models for Scientific Literature Sinha, Aarush Virk, Viraj Chakraborty, Dipshikha Sreeja, P. S. Computation and Language Artificial Intelligence Language Models [LMs] are now playing an increasingly large role in information generation and synthesis; the representation of scientific knowledge in these systems needs to be highly accurate. A prime challenge is hallucination; that is, generating apparently plausible but actually false information, including invented citations and nonexistent research papers. This kind of inaccuracy is dangerous in all the domains that require high levels of factual correctness, such as academia and education. This work presents a pipeline for evaluating the frequency with which language models hallucinate in generating responses in the scientific literature. We propose ArxEval, an evaluation pipeline with two tasks using ArXiv as a repository: Jumbled Titles and Mixed Titles. Our evaluation includes fifteen widely used language models and provides comparative insights into their reliability in handling scientific literature. |
| title | ArxEval: Evaluating Retrieval and Generation in Language Models for Scientific Literature |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.10483 |