Human-AI Collaboration for Estimating Scientific Replicability
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911721626533888 |
|---|---|
| author | Chakravorti, Tatiana Fraleigh, Robert Fritton, Timothy Griffin, Christopher Singh, Vaibhav Koneru, Sai Giles, C. Lee Pennock, David Kwasnica, Anthony Rajtmajer, Sarah |
| author_facet | Chakravorti, Tatiana Fraleigh, Robert Fritton, Timothy Griffin, Christopher Singh, Vaibhav Koneru, Sai Giles, C. Lee Pennock, David Kwasnica, Anthony Rajtmajer, Sarah |
| contents | Determining whether published scientific findings can successfully be replicated is a long-standing challenge in the empirical sciences. Existing approaches for replicability assessment typically rely either on human judgment, i.e., creative assembly of human experts, or on machine learning models trained on paper content metadata. While both approaches have demonstrated value, each also has important limitations. Human forecasts can be influenced by cognitive biases and narrow exposure to the research literature, while automated assessments often struggle to capture contextual cues and subtle signals of credibility. In this paper, we examine a hybrid approach. Specifically, we introduce a hybrid prediction market in which algorithmic agents trade alongside human participants to jointly estimate the likelihood that a published scientific finding will be corroborated via the outcome of a controlled replication study. Agents are trained on outcomes from hundreds of prior replication studies while human participants contribute domain knowledge through real-time trading. We evaluate this hybrid approach through multiple live experiments involving participants from different academic disciplines and compare its performance to artificial-only and human-only baselines. Our results show that, except for a few cases, hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_27394 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Human-AI Collaboration for Estimating Scientific Replicability Chakravorti, Tatiana Fraleigh, Robert Fritton, Timothy Griffin, Christopher Singh, Vaibhav Koneru, Sai Giles, C. Lee Pennock, David Kwasnica, Anthony Rajtmajer, Sarah Computers and Society Artificial Intelligence Human-Computer Interaction Multiagent Systems Determining whether published scientific findings can successfully be replicated is a long-standing challenge in the empirical sciences. Existing approaches for replicability assessment typically rely either on human judgment, i.e., creative assembly of human experts, or on machine learning models trained on paper content metadata. While both approaches have demonstrated value, each also has important limitations. Human forecasts can be influenced by cognitive biases and narrow exposure to the research literature, while automated assessments often struggle to capture contextual cues and subtle signals of credibility. In this paper, we examine a hybrid approach. Specifically, we introduce a hybrid prediction market in which algorithmic agents trade alongside human participants to jointly estimate the likelihood that a published scientific finding will be corroborated via the outcome of a controlled replication study. Agents are trained on outcomes from hundreds of prior replication studies while human participants contribute domain knowledge through real-time trading. We evaluate this hybrid approach through multiple live experiments involving participants from different academic disciplines and compare its performance to artificial-only and human-only baselines. Our results show that, except for a few cases, hybrid markets match or outperform artificial prediction markets, producing more accurate and reliable replication forecasts. |
| title | Human-AI Collaboration for Estimating Scientific Replicability |
| topic | Computers and Society Artificial Intelligence Human-Computer Interaction Multiagent Systems |
| url | https://arxiv.org/abs/2605.27394 |