On the Properties and Estimation of Pointwise Mutual Information Profiles
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917677624197120 |
|---|---|
| author | Czyż, Paweł Grabowski, Frederic Vogt, Julia E. Beerenwinkel, Niko Marx, Alexander |
| author_facet | Czyż, Paweł Grabowski, Frederic Vogt, Julia E. Beerenwinkel, Niko Marx, Alexander |
| contents | The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_10240 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | On the Properties and Estimation of Pointwise Mutual Information Profiles Czyż, Paweł Grabowski, Frederic Vogt, Julia E. Beerenwinkel, Niko Marx, Alexander Machine Learning Information Theory The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary. |
| title | On the Properties and Estimation of Pointwise Mutual Information Profiles |
| topic | Machine Learning Information Theory |
| url | https://arxiv.org/abs/2310.10240 |