Integrating measures of replicability into scholarly search: Challenges and opportunities
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929334426533888 |
|---|---|
| author | Wu, Chuhao Chakravorti, Tatiana Carroll, John Rajtmajer, Sarah |
| author_facet | Wu, Chuhao Chakravorti, Tatiana Carroll, John Rajtmajer, Sarah |
| contents | Challenges to reproducibility and replicability have gained widespread attention, driven by large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into the literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a replicability estimation tool. Our findings suggest that participants tend to confuse replicability with generalizability and related concepts. Information about replicability can support researchers throughout the research design processes. However, the use of AI estimation is debatable due to the lack of explainability and transparency. The ethical implications of AI-enabled confidence assessment must be further studied before such tools could be widely accepted. We discuss implications for the design of technological tools to support scholarly activities and advance replicability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_00653 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Integrating measures of replicability into scholarly search: Challenges and opportunities Wu, Chuhao Chakravorti, Tatiana Carroll, John Rajtmajer, Sarah Digital Libraries Human-Computer Interaction Challenges to reproducibility and replicability have gained widespread attention, driven by large replication projects with lukewarm success rates. A nascent work has emerged developing algorithms to estimate the replicability of published findings. The current study explores ways in which AI-enabled signals of confidence in research might be integrated into the literature search. We interview 17 PhD researchers about their current processes for literature search and ask them to provide feedback on a replicability estimation tool. Our findings suggest that participants tend to confuse replicability with generalizability and related concepts. Information about replicability can support researchers throughout the research design processes. However, the use of AI estimation is debatable due to the lack of explainability and transparency. The ethical implications of AI-enabled confidence assessment must be further studied before such tools could be widely accepted. We discuss implications for the design of technological tools to support scholarly activities and advance replicability. |
| title | Integrating measures of replicability into scholarly search: Challenges and opportunities |
| topic | Digital Libraries Human-Computer Interaction |
| url | https://arxiv.org/abs/2311.00653 |