Normalized Conditional Mutual Information Surrogate Loss for Deep Neural Classifiers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917209006145536 |
|---|---|
| author | Ye, Linfeng Chi, Zhixiang Plataniotis, Konstantinos N. Yang, En-hui |
| author_facet | Ye, Linfeng Chi, Zhixiang Plataniotis, Konstantinos N. Yang, En-hui |
| contents | In this paper, we propose a novel information theoretic surrogate loss; normalized conditional mutual information (NCMI); as a drop in alternative to the de facto cross-entropy (CE) for training deep neural network (DNN) based classifiers. We first observe that the model's NCMI is inversely proportional to its accuracy. Building on this insight, we introduce an alternating algorithm to efficiently minimize the NCMI. Across image recognition and whole-slide imaging (WSI) subtyping benchmarks, NCMI-trained models surpass state of the art losses by substantial margins at a computational cost comparable to that of CE. Notably, on ImageNet, NCMI yields a 2.77% top-1 accuracy improvement with ResNet-50 comparing to the CE; on CAMELYON-17, replacing CE with NCMI improves the macro-F1 by 8.6% over the strongest baseline. Gains are consistent across various architectures and batch sizes, suggesting that NCMI is a practical and competitive alternative to CE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02543 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Normalized Conditional Mutual Information Surrogate Loss for Deep Neural Classifiers Ye, Linfeng Chi, Zhixiang Plataniotis, Konstantinos N. Yang, En-hui Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Information Theory In this paper, we propose a novel information theoretic surrogate loss; normalized conditional mutual information (NCMI); as a drop in alternative to the de facto cross-entropy (CE) for training deep neural network (DNN) based classifiers. We first observe that the model's NCMI is inversely proportional to its accuracy. Building on this insight, we introduce an alternating algorithm to efficiently minimize the NCMI. Across image recognition and whole-slide imaging (WSI) subtyping benchmarks, NCMI-trained models surpass state of the art losses by substantial margins at a computational cost comparable to that of CE. Notably, on ImageNet, NCMI yields a 2.77% top-1 accuracy improvement with ResNet-50 comparing to the CE; on CAMELYON-17, replacing CE with NCMI improves the macro-F1 by 8.6% over the strongest baseline. Gains are consistent across various architectures and batch sizes, suggesting that NCMI is a practical and competitive alternative to CE. |
| title | Normalized Conditional Mutual Information Surrogate Loss for Deep Neural Classifiers |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Information Theory |
| url | https://arxiv.org/abs/2601.02543 |