Better Together: Cross and Joint Covariances Enhance Signal Detectability in Undersampled Data
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908937795665920 |
|---|---|
| author | Swain, Arabind Ridout, Sean Alexander Nemenman, Ilya |
| author_facet | Swain, Arabind Ridout, Sean Alexander Nemenman, Ilya |
| contents | Many data-science applications involve detecting a shared signal between two high-dimensional variables. Using random matrix theory methods, we determine when such signal can be detected and reconstructed from sample correlations, despite the background of sampling noise induced correlations. We consider three different covariance matrices constructed from two high-dimensional variables: their individual self covariance, their cross covariance, and the self covariance of the concatenated (joint) variable, which incorporates the self and the cross correlation blocks. We observe the expected Baik, Ben Arous, and Péché detectability phase transition in all these covariance matrices, and we show that joint and cross covariance matrices always reconstruct the shared signal earlier than the self covariances. Whether the joint or the cross approach is better depends on the mismatch of dimensionalities between the variables. We discuss what these observations mean for choosing the right method for detecting linear correlations in data and how these findings may generalize to nonlinear statistical dependencies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_22207 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Better Together: Cross and Joint Covariances Enhance Signal Detectability in Undersampled Data Swain, Arabind Ridout, Sean Alexander Nemenman, Ilya Disordered Systems and Neural Networks Machine Learning Data Analysis, Statistics and Probability Many data-science applications involve detecting a shared signal between two high-dimensional variables. Using random matrix theory methods, we determine when such signal can be detected and reconstructed from sample correlations, despite the background of sampling noise induced correlations. We consider three different covariance matrices constructed from two high-dimensional variables: their individual self covariance, their cross covariance, and the self covariance of the concatenated (joint) variable, which incorporates the self and the cross correlation blocks. We observe the expected Baik, Ben Arous, and Péché detectability phase transition in all these covariance matrices, and we show that joint and cross covariance matrices always reconstruct the shared signal earlier than the self covariances. Whether the joint or the cross approach is better depends on the mismatch of dimensionalities between the variables. We discuss what these observations mean for choosing the right method for detecting linear correlations in data and how these findings may generalize to nonlinear statistical dependencies. |
| title | Better Together: Cross and Joint Covariances Enhance Signal Detectability in Undersampled Data |
| topic | Disordered Systems and Neural Networks Machine Learning Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2507.22207 |