Tighter Bounds on the Information Bottleneck with Application to Deep Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909102651736064 |
|---|---|
| author | Weingarten, Nir Yakhini, Zohar Butman, Moshe Gilad-Bachrach, Ran |
| author_facet | Weingarten, Nir Yakhini, Zohar Butman, Moshe Gilad-Bachrach, Ran |
| contents | Deep Neural Nets (DNNs) learn latent representations induced by their downstream task, objective function, and other parameters. The quality of the learned representations impacts the DNN's generalization ability and the coherence of the emerging latent space. The Information Bottleneck (IB) provides a hypothetically optimal framework for data modeling, yet it is often intractable. Recent efforts combined DNNs with the IB by applying VAE-inspired variational methods to approximate bounds on mutual information, resulting in improved robustness to adversarial attacks. This work introduces a new and tighter variational bound for the IB, improving performance of previous IB-inspired DNNs. These advancements strengthen the case for the IB and its variational approximations as a data modeling framework, and provide a simple method to significantly enhance the adversarial robustness of classifier DNNs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07639 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tighter Bounds on the Information Bottleneck with Application to Deep Learning Weingarten, Nir Yakhini, Zohar Butman, Moshe Gilad-Bachrach, Ran Machine Learning Artificial Intelligence Information Theory 94A08, 94A10, 94A11, 68T06, 62B04, 62B08 I.2; E.4; I.4; I.7 Deep Neural Nets (DNNs) learn latent representations induced by their downstream task, objective function, and other parameters. The quality of the learned representations impacts the DNN's generalization ability and the coherence of the emerging latent space. The Information Bottleneck (IB) provides a hypothetically optimal framework for data modeling, yet it is often intractable. Recent efforts combined DNNs with the IB by applying VAE-inspired variational methods to approximate bounds on mutual information, resulting in improved robustness to adversarial attacks. This work introduces a new and tighter variational bound for the IB, improving performance of previous IB-inspired DNNs. These advancements strengthen the case for the IB and its variational approximations as a data modeling framework, and provide a simple method to significantly enhance the adversarial robustness of classifier DNNs. |
| title | Tighter Bounds on the Information Bottleneck with Application to Deep Learning |
| topic | Machine Learning Artificial Intelligence Information Theory 94A08, 94A10, 94A11, 68T06, 62B04, 62B08 I.2; E.4; I.4; I.7 |
| url | https://arxiv.org/abs/2402.07639 |