Large Scale Partial Correlation Screening with Uncertainty Quantification
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918145427505152 |
|---|---|
| author | Neo, Emily Radchenko, Peter Rajaratnam, Bala |
| author_facet | Neo, Emily Radchenko, Peter Rajaratnam, Bala |
| contents | Identifying multivariate dependencies in high-dimensional data is an important problem in large-scale inference. This problem has motivated recent advances in mining (partial) correlations, which focus on the challenging ultra-high dimensional setting where the sample size, n, is fixed, while the number of features, p, grows without bound. The state-of-the-art method for partial correlation screening can lead to undesirable results. This paper introduces a novel principled framework for partial correlation screening with error control (PARSEC), which leverages the connection between partial correlations and regression coefficients. We establish the inferential properties of PARSEC when n is fixed and p grows super-exponentially. First, we provide "fixed-n-large-p" asymptotic expressions for the familywise error rate (FWER) and k-FWER. Equally importantly, our analysis leads to a novel discovery which permits the calculation of exact marginal p-values for controlling the false discovery rate (FDR), and also the positive FDR (pFDR). To our knowledge, no other competing approach in the "fixed-n large-p" setting allows for error control across the spectrum of multiple hypothesis testing metrics. We establish the computational complexity of PARSEC and rigorously demonstrate its scalability to the large p setting. The theory and methods are successfully validated on simulated and real data, and PARSEC is shown to outperform the current state-of-the-art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17128 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Large Scale Partial Correlation Screening with Uncertainty Quantification Neo, Emily Radchenko, Peter Rajaratnam, Bala Methodology Identifying multivariate dependencies in high-dimensional data is an important problem in large-scale inference. This problem has motivated recent advances in mining (partial) correlations, which focus on the challenging ultra-high dimensional setting where the sample size, n, is fixed, while the number of features, p, grows without bound. The state-of-the-art method for partial correlation screening can lead to undesirable results. This paper introduces a novel principled framework for partial correlation screening with error control (PARSEC), which leverages the connection between partial correlations and regression coefficients. We establish the inferential properties of PARSEC when n is fixed and p grows super-exponentially. First, we provide "fixed-n-large-p" asymptotic expressions for the familywise error rate (FWER) and k-FWER. Equally importantly, our analysis leads to a novel discovery which permits the calculation of exact marginal p-values for controlling the false discovery rate (FDR), and also the positive FDR (pFDR). To our knowledge, no other competing approach in the "fixed-n large-p" setting allows for error control across the spectrum of multiple hypothesis testing metrics. We establish the computational complexity of PARSEC and rigorously demonstrate its scalability to the large p setting. The theory and methods are successfully validated on simulated and real data, and PARSEC is shown to outperform the current state-of-the-art. |
| title | Large Scale Partial Correlation Screening with Uncertainty Quantification |
| topic | Methodology |
| url | https://arxiv.org/abs/2509.17128 |