Shannon invariants: A scalable approach to information decomposition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916702140235776 |
|---|---|
| author | Gutknecht, Aaron J. Rosas, Fernando E. Ehrlich, David A. Makkeh, Abdullah Mediano, Pedro A. M. Wibral, Michael |
| author_facet | Gutknecht, Aaron J. Rosas, Fernando E. Ehrlich, David A. Makkeh, Abdullah Mediano, Pedro A. M. Wibral, Michael |
| contents | Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order patterns with distinct properties across scales. Investigating how these systems process information remains challenging due to difficulties in defining appropriate multivariate metrics and ensuring their scalability to large systems. To address these challenges, we introduce a novel framework based on what we call "Shannon invariants" -- quantities that capture essential properties of high-order information processing in a way that depends only on the definition of entropy and can be efficiently calculated for large systems. Our theoretical results demonstrate how Shannon invariants can be used to resolve long-standing ambiguities regarding the interpretation of widely used multivariate information-theoretic measures. Moreover, our practical results reveal distinctive information-processing signatures of various deep learning architectures across layers, which lead to new insights into how these systems process information and how this evolves during training. Overall, our framework resolves fundamental limitations in analyzing high-order phenomena and offers broad opportunities for theoretical developments and empirical analyses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15779 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Shannon invariants: A scalable approach to information decomposition Gutknecht, Aaron J. Rosas, Fernando E. Ehrlich, David A. Makkeh, Abdullah Mediano, Pedro A. M. Wibral, Michael Information Theory Artificial Intelligence Machine Learning Adaptation and Self-Organizing Systems Data Analysis, Statistics and Probability Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order patterns with distinct properties across scales. Investigating how these systems process information remains challenging due to difficulties in defining appropriate multivariate metrics and ensuring their scalability to large systems. To address these challenges, we introduce a novel framework based on what we call "Shannon invariants" -- quantities that capture essential properties of high-order information processing in a way that depends only on the definition of entropy and can be efficiently calculated for large systems. Our theoretical results demonstrate how Shannon invariants can be used to resolve long-standing ambiguities regarding the interpretation of widely used multivariate information-theoretic measures. Moreover, our practical results reveal distinctive information-processing signatures of various deep learning architectures across layers, which lead to new insights into how these systems process information and how this evolves during training. Overall, our framework resolves fundamental limitations in analyzing high-order phenomena and offers broad opportunities for theoretical developments and empirical analyses. |
| title | Shannon invariants: A scalable approach to information decomposition |
| topic | Information Theory Artificial Intelligence Machine Learning Adaptation and Self-Organizing Systems Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2504.15779 |