Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915275500158976 |
|---|---|
| author | Camilli, Francesco Tieplova, Daria Bergamin, Eleonora Barbier, Jean |
| author_facet | Camilli, Francesco Tieplova, Daria Bergamin, Eleonora Barbier, Jean |
| contents | We rigorously analyse fully-trained neural networks of arbitrary depth in the Bayesian optimal setting in the so-called proportional scaling regime where the number of training samples and width of the input and all inner layers diverge proportionally. We prove an information-theoretic equivalence between the Bayesian deep neural network model trained from data generated by a teacher with matching architecture, and a simpler model of optimal inference in a generalized linear model. This equivalence enables us to compute the optimal generalization error for deep neural networks in this regime. We thus prove the "deep Gaussian equivalence principle" conjectured in Cui et al. (2023) (arXiv:2302.00375). Our result highlights that in order to escape this "trivialisation" of deep neural networks (in the sense of reduction to a linear model) happening in the strongly overparametrized proportional regime, models trained from much more data have to be considered. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03577 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime Camilli, Francesco Tieplova, Daria Bergamin, Eleonora Barbier, Jean Statistics Theory Disordered Systems and Neural Networks Information Theory Mathematical Physics Machine Learning We rigorously analyse fully-trained neural networks of arbitrary depth in the Bayesian optimal setting in the so-called proportional scaling regime where the number of training samples and width of the input and all inner layers diverge proportionally. We prove an information-theoretic equivalence between the Bayesian deep neural network model trained from data generated by a teacher with matching architecture, and a simpler model of optimal inference in a generalized linear model. This equivalence enables us to compute the optimal generalization error for deep neural networks in this regime. We thus prove the "deep Gaussian equivalence principle" conjectured in Cui et al. (2023) (arXiv:2302.00375). Our result highlights that in order to escape this "trivialisation" of deep neural networks (in the sense of reduction to a linear model) happening in the strongly overparametrized proportional regime, models trained from much more data have to be considered. |
| title | Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime |
| topic | Statistics Theory Disordered Systems and Neural Networks Information Theory Mathematical Physics Machine Learning |
| url | https://arxiv.org/abs/2505.03577 |