Universality laws for Gaussian mixtures in generalized linear models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916329833889792 |
|---|---|
| author | Dandi, Yatin Stephan, Ludovic Krzakala, Florent Loureiro, Bruno Zdeborová, Lenka |
| author_facet | Dandi, Yatin Stephan, Ludovic Krzakala, Florent Loureiro, Bruno Zdeborová, Lenka |
| contents | Let $(x_{i}, y_{i})_{i=1,\dots,n}$ denote independent samples from a general mixture distribution $\sum_{c\in\mathcal{C}}ρ_{c}P_{c}^{x}$, and consider the hypothesis class of generalized linear models $\hat{y} = F(Θ^{\top}x)$. In this work, we investigate the asymptotic joint statistics of the family of generalized linear estimators $(Θ_{1}, \dots, Θ_{M})$ obtained either from (a) minimizing an empirical risk $\hat{R}_{n}(Θ;X,y)$ or (b) sampling from the associated Gibbs measure $\exp(-βn \hat{R}_{n}(Θ;X,y))$. Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution $P_{c}^{x}$. In particular, this allow us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results to different machine learning tasks of interest, such as ensembling and uncertainty |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_08933 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Universality laws for Gaussian mixtures in generalized linear models Dandi, Yatin Stephan, Ludovic Krzakala, Florent Loureiro, Bruno Zdeborová, Lenka Statistics Theory Machine Learning Let $(x_{i}, y_{i})_{i=1,\dots,n}$ denote independent samples from a general mixture distribution $\sum_{c\in\mathcal{C}}ρ_{c}P_{c}^{x}$, and consider the hypothesis class of generalized linear models $\hat{y} = F(Θ^{\top}x)$. In this work, we investigate the asymptotic joint statistics of the family of generalized linear estimators $(Θ_{1}, \dots, Θ_{M})$ obtained either from (a) minimizing an empirical risk $\hat{R}_{n}(Θ;X,y)$ or (b) sampling from the associated Gibbs measure $\exp(-βn \hat{R}_{n}(Θ;X,y))$. Our main contribution is to characterize under which conditions the asymptotic joint statistics of this family depends (on a weak sense) only on the means and covariances of the class conditional features distribution $P_{c}^{x}$. In particular, this allow us to prove the universality of different quantities of interest, such as the training and generalization errors, redeeming a recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis. Finally, we discuss the applications of our results to different machine learning tasks of interest, such as ensembling and uncertainty |
| title | Universality laws for Gaussian mixtures in generalized linear models |
| topic | Statistics Theory Machine Learning |
| url | https://arxiv.org/abs/2302.08933 |