Beyond Arbitrary Replications: A Principled Approach to Simulation Design in Causal Inference
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915299295494144 |
|---|---|
| author | Souto, Hugo Gobato Neto, Francisco Louzada |
| author_facet | Souto, Hugo Gobato Neto, Francisco Louzada |
| contents | Evaluation of novel treatment effect estimators frequently relies on simulation studies lacking formal statistical comparisons and using arbitrary numbers of replications ($J$). This hinders reproducibility and efficiency. We propose the Test-Informed Simulation Count Algorithm (TISCA) to address these shortcomings. TISCA integrates Welch's t-tests with power analysis, iteratively running simulations until a pre-specified power (e.g., 0.8) is achieved for detecting a user-defined minimum detectable effect size (MDE) at a given significance level ($α$). This yields a statistically justified simulation count ($J$) and rigorous model comparisons. Our bibliometric study confirms the heterogeneity of current practices regarding $J$. A case study revisiting McJames et al. (2024) demonstrates TISCA identifies sufficient simulations ($J=500$ vs. original $J=1000$), saving computational resources while providing statistically sound evidence. TISCA promotes rigorous, efficient, and sustainable simulation practices in causal inference and beyond. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05161 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Arbitrary Replications: A Principled Approach to Simulation Design in Causal Inference Souto, Hugo Gobato Neto, Francisco Louzada Methodology Evaluation of novel treatment effect estimators frequently relies on simulation studies lacking formal statistical comparisons and using arbitrary numbers of replications ($J$). This hinders reproducibility and efficiency. We propose the Test-Informed Simulation Count Algorithm (TISCA) to address these shortcomings. TISCA integrates Welch's t-tests with power analysis, iteratively running simulations until a pre-specified power (e.g., 0.8) is achieved for detecting a user-defined minimum detectable effect size (MDE) at a given significance level ($α$). This yields a statistically justified simulation count ($J$) and rigorous model comparisons. Our bibliometric study confirms the heterogeneity of current practices regarding $J$. A case study revisiting McJames et al. (2024) demonstrates TISCA identifies sufficient simulations ($J=500$ vs. original $J=1000$), saving computational resources while providing statistically sound evidence. TISCA promotes rigorous, efficient, and sustainable simulation practices in causal inference and beyond. |
| title | Beyond Arbitrary Replications: A Principled Approach to Simulation Design in Causal Inference |
| topic | Methodology |
| url | https://arxiv.org/abs/2409.05161 |