Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.08222 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915246881374208 |
|---|---|
| author | Yang, Hanzhe Wu, Youlong Wen, Dingzhu Zhou, Yong Shi, Yuanming |
| author_facet | Yang, Hanzhe Wu, Youlong Wen, Dingzhu Zhou, Yong Shi, Yuanming |
| contents | The Information Bottleneck (IB) principle has emerged as a promising approach for enhancing the generalization, robustness, and interpretability of deep neural networks, demonstrating efficacy across image segmentation, document clustering, and semantic communication. Among IB implementations, the IB Lagrangian method, employing Lagrangian multipliers, is widely adopted. While numerous methods for the optimizations of IB Lagrangian based on variational bounds and neural estimators are feasible, their performance is highly dependent on the quality of their design, which is inherently prone to errors. To address this limitation, we introduce Structured IB, a framework for investigating potential structured features. By incorporating auxiliary encoders to extract missing informative features, we generate more informative representations. Our experiments demonstrate superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08222 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Structured IB: Improving Information Bottleneck with Structured Feature Learning Yang, Hanzhe Wu, Youlong Wen, Dingzhu Zhou, Yong Shi, Yuanming Information Theory Machine Learning The Information Bottleneck (IB) principle has emerged as a promising approach for enhancing the generalization, robustness, and interpretability of deep neural networks, demonstrating efficacy across image segmentation, document clustering, and semantic communication. Among IB implementations, the IB Lagrangian method, employing Lagrangian multipliers, is widely adopted. While numerous methods for the optimizations of IB Lagrangian based on variational bounds and neural estimators are feasible, their performance is highly dependent on the quality of their design, which is inherently prone to errors. To address this limitation, we introduce Structured IB, a framework for investigating potential structured features. By incorporating auxiliary encoders to extract missing informative features, we generate more informative representations. Our experiments demonstrate superior prediction accuracy and task-relevant information preservation compared to the original IB Lagrangian method, even with reduced network size. |
| title | Structured IB: Improving Information Bottleneck with Structured Feature Learning |
| topic | Information Theory Machine Learning |
| url | https://arxiv.org/abs/2412.08222 |