Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911278659796992 |
|---|---|
| author | Bartoš, František Hoogeveen, Suzanne Sarafoglou, Alexandra Pawel, Samuel |
| author_facet | Bartoš, František Hoogeveen, Suzanne Sarafoglou, Alexandra Pawel, Samuel |
| contents | Empirical claims often rely on one population, design, and analysis. Many-analysts, multiverse, and robustness studies expose how results can vary across plausible analytic choices. Synthesizing these results, however, is nontrivial as all results are computed from the same dataset. We introduce single-dataset meta-analysis, a weighted-likelihood approach that incorporates the information in the dataset at most once. It prevents overconfident inferences that would arise if a standard meta-analysis was applied to the data. Single-dataset meta-analysis yields meta-analytic point and interval estimates of the average effect across analytic approaches and of between-analyst heterogeneity, and can be supplied by classical and Bayesian hypothesis tests. Both the common-effect and random-effects versions of the model can be estimated by standard meta-analytic software with small input adjustments. We demonstrate the method via application to the many-analysts study on racial bias in soccer, the many-analysts study of marital status and cardiovascular disease, and the multiverse study on technology use and well-being. The results show how single-dataset meta-analysis complements the qualitative evaluation of many-analysts and multiverse studies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_17064 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies Bartoš, František Hoogeveen, Suzanne Sarafoglou, Alexandra Pawel, Samuel Methodology 62A99 G.3 Empirical claims often rely on one population, design, and analysis. Many-analysts, multiverse, and robustness studies expose how results can vary across plausible analytic choices. Synthesizing these results, however, is nontrivial as all results are computed from the same dataset. We introduce single-dataset meta-analysis, a weighted-likelihood approach that incorporates the information in the dataset at most once. It prevents overconfident inferences that would arise if a standard meta-analysis was applied to the data. Single-dataset meta-analysis yields meta-analytic point and interval estimates of the average effect across analytic approaches and of between-analyst heterogeneity, and can be supplied by classical and Bayesian hypothesis tests. Both the common-effect and random-effects versions of the model can be estimated by standard meta-analytic software with small input adjustments. We demonstrate the method via application to the many-analysts study on racial bias in soccer, the many-analysts study of marital status and cardiovascular disease, and the multiverse study on technology use and well-being. The results show how single-dataset meta-analysis complements the qualitative evaluation of many-analysts and multiverse studies. |
| title | Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies |
| topic | Methodology 62A99 G.3 |
| url | https://arxiv.org/abs/2511.17064 |