Information-theoretic Quantification of High-order Feature Effects in Classification Problems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911041713078272 |
|---|---|
| author | Lazic, Ivan Barà, Chiara Iovino, Marta Stramaglia, Sebastiano Jakovljevic, Niksa Faes, Luca |
| author_facet | Lazic, Ivan Barà, Chiara Iovino, Marta Stramaglia, Sebastiano Jakovljevic, Niksa Faes, Luca |
| contents | Understanding the contribution of individual features in predictive models remains a central goal in interpretable machine learning, and while many model-agnostic methods exist to estimate feature importance, they often fall short in capturing high-order interactions and disentangling overlapping contributions. In this work, we present an information-theoretic extension of the High-order interactions for Feature importance (Hi-Fi) method, leveraging Conditional Mutual Information (CMI) estimated via a k-Nearest Neighbor (kNN) approach working on mixed discrete and continuous random variables. Our framework decomposes feature contributions into unique, synergistic, and redundant components, offering a richer, model-independent understanding of their predictive roles. We validate the method using synthetic datasets with known Gaussian structures, where ground truth interaction patterns are analytically derived, and further test it on non-Gaussian and real-world gene expression data from TCGA-BRCA. Results indicate that the proposed estimator accurately recovers theoretical and expected findings, providing a potential use case for developing feature selection algorithms or model development based on interaction analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04362 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Information-theoretic Quantification of High-order Feature Effects in Classification Problems Lazic, Ivan Barà, Chiara Iovino, Marta Stramaglia, Sebastiano Jakovljevic, Niksa Faes, Luca Machine Learning Data Analysis, Statistics and Probability Understanding the contribution of individual features in predictive models remains a central goal in interpretable machine learning, and while many model-agnostic methods exist to estimate feature importance, they often fall short in capturing high-order interactions and disentangling overlapping contributions. In this work, we present an information-theoretic extension of the High-order interactions for Feature importance (Hi-Fi) method, leveraging Conditional Mutual Information (CMI) estimated via a k-Nearest Neighbor (kNN) approach working on mixed discrete and continuous random variables. Our framework decomposes feature contributions into unique, synergistic, and redundant components, offering a richer, model-independent understanding of their predictive roles. We validate the method using synthetic datasets with known Gaussian structures, where ground truth interaction patterns are analytically derived, and further test it on non-Gaussian and real-world gene expression data from TCGA-BRCA. Results indicate that the proposed estimator accurately recovers theoretical and expected findings, providing a potential use case for developing feature selection algorithms or model development based on interaction analysis. |
| title | Information-theoretic Quantification of High-order Feature Effects in Classification Problems |
| topic | Machine Learning Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2507.04362 |