Statistical divergences in high-dimensional hypothesis testing and a modern technique for estimating them
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910550068297728 |
|---|---|
| author | Wilkinson, Jeremy J. H. Lester, Christopher G. |
| author_facet | Wilkinson, Jeremy J. H. Lester, Christopher G. |
| contents | Hypothesis testing in high dimensional data is a notoriously difficult problem without direct access to competing models' likelihood functions. This paper argues that statistical divergences can be used to quantify the difference between the population distributions of observed data and competing models, justifying their use as the basis of a hypothesis test. We go on to point out how modern techniques for functional optimization let us estimate many divergences, without the need for population likelihood functions, using samples from two distributions alone. We use a physics-based example to show how the proposed two-sample test can be implemented in practice, and discuss the necessary steps required to mature the ideas presented into an experimental framework. The code used has been made available for others to use. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_06397 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Statistical divergences in high-dimensional hypothesis testing and a modern technique for estimating them Wilkinson, Jeremy J. H. Lester, Christopher G. Data Analysis, Statistics and Probability High Energy Physics - Experiment High Energy Physics - Phenomenology Statistics Theory Hypothesis testing in high dimensional data is a notoriously difficult problem without direct access to competing models' likelihood functions. This paper argues that statistical divergences can be used to quantify the difference between the population distributions of observed data and competing models, justifying their use as the basis of a hypothesis test. We go on to point out how modern techniques for functional optimization let us estimate many divergences, without the need for population likelihood functions, using samples from two distributions alone. We use a physics-based example to show how the proposed two-sample test can be implemented in practice, and discuss the necessary steps required to mature the ideas presented into an experimental framework. The code used has been made available for others to use. |
| title | Statistical divergences in high-dimensional hypothesis testing and a modern technique for estimating them |
| topic | Data Analysis, Statistics and Probability High Energy Physics - Experiment High Energy Physics - Phenomenology Statistics Theory |
| url | https://arxiv.org/abs/2405.06397 |