Mutual Information Multinomial Estimation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929463382507520 |
|---|---|
| author | Chen, Yanzhi Ou, Zijing Weller, Adrian Li, Yingzhen |
| author_facet | Chen, Yanzhi Ou, Zijing Weller, Adrian Li, Yingzhen |
| contents | Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. This work proposes a new estimator for mutual information. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. This preliminary estimate serves as a bridge between the joint and the marginal distribution, and by comparing with this bridge distribution we can easily obtain the true difference between the joint distributions and the marginal distributions. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09377 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mutual Information Multinomial Estimation Chen, Yanzhi Ou, Zijing Weller, Adrian Li, Yingzhen Machine Learning Information Theory Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. This work proposes a new estimator for mutual information. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. This preliminary estimate serves as a bridge between the joint and the marginal distribution, and by comparing with this bridge distribution we can easily obtain the true difference between the joint distributions and the marginal distributions. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method. |
| title | Mutual Information Multinomial Estimation |
| topic | Machine Learning Information Theory |
| url | https://arxiv.org/abs/2408.09377 |