Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.13671 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913896128839680 |
|---|---|
| author | Huang, Danyang Wang, Liyuan Zhu, Liping |
| author_facet | Huang, Danyang Wang, Liyuan Zhu, Liping |
| contents | It is generally believed that more observations provide more information. However, we observe that in the independence test for rare events, the power of the test is, surprisingly, determined by the number of rare events rather than the total sample size. Moreover, the correlations tend to shrink to zero even as the total sample size increases, as long as the proportion of rare events decreases. We demonstrate this phenomenon in both fixed and high-dimensional settings. To address these issues, we first rescale the covariances to account for the presence of rare events. We then propose a boosted procedure that uses only a small subset of non-rare events, yet achieves nearly the same power as using the full set of observations. As a result, computational complexity is significantly reduced. The theoretical properties, including asymptotic distribution and local power analysis, are carefully derived for both the rescaled statistic based on the full sample and the boosted test statistic based on subsampling. Furthermore, we extend the theory to multi-class rare events. Extensive simulations and real-world data analyses confirm the effectiveness and computational efficiency of the proposed approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_13671 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do more observations bring more information in rare events? Huang, Danyang Wang, Liyuan Zhu, Liping Methodology It is generally believed that more observations provide more information. However, we observe that in the independence test for rare events, the power of the test is, surprisingly, determined by the number of rare events rather than the total sample size. Moreover, the correlations tend to shrink to zero even as the total sample size increases, as long as the proportion of rare events decreases. We demonstrate this phenomenon in both fixed and high-dimensional settings. To address these issues, we first rescale the covariances to account for the presence of rare events. We then propose a boosted procedure that uses only a small subset of non-rare events, yet achieves nearly the same power as using the full set of observations. As a result, computational complexity is significantly reduced. The theoretical properties, including asymptotic distribution and local power analysis, are carefully derived for both the rescaled statistic based on the full sample and the boosted test statistic based on subsampling. Furthermore, we extend the theory to multi-class rare events. Extensive simulations and real-world data analyses confirm the effectiveness and computational efficiency of the proposed approach. |
| title | Do more observations bring more information in rare events? |
| topic | Methodology |
| url | https://arxiv.org/abs/2506.13671 |