Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.02306 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929750022291456 |
|---|---|
| author | Hemerik, Jesse Koning, Nick W |
| author_facet | Hemerik, Jesse Koning, Nick W |
| contents | A fundamental assumption of classical hypothesis testing is that the significance threshold $α$ is chosen independently from the data. The validity of confidence intervals likewise relies on choosing $α$ beforehand. We point out that the independence of $α$ is guaranteed in practice because, in most fields, there exists one standard $α$ that everyone uses -- so that $α$ is automatically independent of everything. However, there have been recent calls to decrease $α$ from $0.05$ to $0.005$. We note that this may lead to multiple accepted standard thresholds within one scientific field. For example, different journals may require different significance thresholds. As a consequence, some researchers may be tempted to conveniently choose their $α$ based on their p-value. We use examples to illustrate that this severely invalidates hypothesis tests, and mention some potential solutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02306 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Choosing alpha post hoc: the danger of multiple standard significance thresholds Hemerik, Jesse Koning, Nick W Applications Methodology 62A01 A fundamental assumption of classical hypothesis testing is that the significance threshold $α$ is chosen independently from the data. The validity of confidence intervals likewise relies on choosing $α$ beforehand. We point out that the independence of $α$ is guaranteed in practice because, in most fields, there exists one standard $α$ that everyone uses -- so that $α$ is automatically independent of everything. However, there have been recent calls to decrease $α$ from $0.05$ to $0.005$. We note that this may lead to multiple accepted standard thresholds within one scientific field. For example, different journals may require different significance thresholds. As a consequence, some researchers may be tempted to conveniently choose their $α$ based on their p-value. We use examples to illustrate that this severely invalidates hypothesis tests, and mention some potential solutions. |
| title | Choosing alpha post hoc: the danger of multiple standard significance thresholds |
| topic | Applications Methodology 62A01 |
| url | https://arxiv.org/abs/2410.02306 |