BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912598079832064 |
|---|---|
| author | Azher, Ibrahim Al Mokarrama, Miftahul Jannat Guo, Zhishuai Choudhury, Sagnik Ray Alhoori, Hamed |
| author_facet | Azher, Ibrahim Al Mokarrama, Miftahul Jannat Guo, Zhishuai Choudhury, Sagnik Ray Alhoori, Hamed |
| contents | In scientific research, ``limitations'' refer to the shortcomings, constraints, or weaknesses of a study. A transparent reporting of such limitations can enhance the quality and reproducibility of research and improve public trust in science. However, authors often underreport limitations in their papers and rely on hedging strategies to meet editorial requirements at the expense of readers' clarity and confidence. This tendency, combined with the surge in scientific publications, has created a pressing need for automated approaches to extract and generate limitations from scholarly papers. To address this need, we present a full architecture for computational analysis of research limitations. Specifically, we (1) create a dataset of limitations from ACL, NeurIPS, and PeerJ papers by extracting them from the text and supplementing them with external reviews; (2) we propose methods to automatically generate limitations using a novel Retrieval Augmented Generation (RAG) technique; (3) we design a fine-grained evaluation framework for generated limitations, along with a meta-evaluation of these techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18207 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text Azher, Ibrahim Al Mokarrama, Miftahul Jannat Guo, Zhishuai Choudhury, Sagnik Ray Alhoori, Hamed Digital Libraries Machine Learning 68T50 Natural language processing In scientific research, ``limitations'' refer to the shortcomings, constraints, or weaknesses of a study. A transparent reporting of such limitations can enhance the quality and reproducibility of research and improve public trust in science. However, authors often underreport limitations in their papers and rely on hedging strategies to meet editorial requirements at the expense of readers' clarity and confidence. This tendency, combined with the surge in scientific publications, has created a pressing need for automated approaches to extract and generate limitations from scholarly papers. To address this need, we present a full architecture for computational analysis of research limitations. Specifically, we (1) create a dataset of limitations from ACL, NeurIPS, and PeerJ papers by extracting them from the text and supplementing them with external reviews; (2) we propose methods to automatically generate limitations using a novel Retrieval Augmented Generation (RAG) technique; (3) we design a fine-grained evaluation framework for generated limitations, along with a meta-evaluation of these techniques. |
| title | BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text |
| topic | Digital Libraries Machine Learning 68T50 Natural language processing |
| url | https://arxiv.org/abs/2505.18207 |