Use as Directed? A Comparison of Software Tools Intended to Check Rigor and Transparency of Published Work
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909703205814272 |
|---|---|
| author | Eckmann, Peter Barnett, Adrian Bannach-Brown, Alexandra Atria, Elisa Pilar Bascunan Cabanac, Guillaume Franzen, Louise Delwen Owen Gazda, Małgorzata Anna Hair, Kaitlyn Howison, James Kilicoglu, Halil Labbe, Cyril McCann, Sarah Nachev, Vladislav Roelandse, Martijn Salholz-Hillel, Maia Schulz, Robert ter Riet, Gerben Vorland, Colby Bandrowski, Anita Weissgerber, Tracey |
| author_facet | Eckmann, Peter Barnett, Adrian Bannach-Brown, Alexandra Atria, Elisa Pilar Bascunan Cabanac, Guillaume Franzen, Louise Delwen Owen Gazda, Małgorzata Anna Hair, Kaitlyn Howison, James Kilicoglu, Halil Labbe, Cyril McCann, Sarah Nachev, Vladislav Roelandse, Martijn Salholz-Hillel, Maia Schulz, Robert ter Riet, Gerben Vorland, Colby Bandrowski, Anita Weissgerber, Tracey |
| contents | The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at https://github.com/PeterEckmann1/tool-comparison. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_17991 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Use as Directed? A Comparison of Software Tools Intended to Check Rigor and Transparency of Published Work Eckmann, Peter Barnett, Adrian Bannach-Brown, Alexandra Atria, Elisa Pilar Bascunan Cabanac, Guillaume Franzen, Louise Delwen Owen Gazda, Małgorzata Anna Hair, Kaitlyn Howison, James Kilicoglu, Halil Labbe, Cyril McCann, Sarah Nachev, Vladislav Roelandse, Martijn Salholz-Hillel, Maia Schulz, Robert ter Riet, Gerben Vorland, Colby Bandrowski, Anita Weissgerber, Tracey Software Engineering Information Retrieval The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at https://github.com/PeterEckmann1/tool-comparison. |
| title | Use as Directed? A Comparison of Software Tools Intended to Check Rigor and Transparency of Published Work |
| topic | Software Engineering Information Retrieval |
| url | https://arxiv.org/abs/2507.17991 |