Gaussian Universality of Perceptrons with Random Labels
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911961002803200 |
|---|---|
| author | Gerace, Federica Krzakala, Florent Loureiro, Bruno Stephan, Ludovic Zdeborová, Lenka |
| author_facet | Gerace, Federica Krzakala, Florent Loureiro, Bruno Stephan, Ludovic Zdeborová, Lenka |
| contents | While classical in many theoretical settings - and in particular in statistical physics-inspired works - the assumption of Gaussian i.i.d. input data is often perceived as a strong limitation in the context of statistics and machine learning. In this study, we redeem this line of work in the case of generalized linear classification, a.k.a. the perceptron model, with random labels. We argue that there is a large universality class of high-dimensional input data for which we obtain the same minimum training loss as for Gaussian data with corresponding data covariance. In the limit of vanishing regularization, we further demonstrate that the training loss is independent of the data covariance. On the theoretical side, we prove this universality for an arbitrary mixture of homogeneous Gaussian clouds. Empirically, we show that the universality holds also for a broad range of real datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2205_13303 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Gaussian Universality of Perceptrons with Random Labels Gerace, Federica Krzakala, Florent Loureiro, Bruno Stephan, Ludovic Zdeborová, Lenka Machine Learning Disordered Systems and Neural Networks Probability Statistics Theory While classical in many theoretical settings - and in particular in statistical physics-inspired works - the assumption of Gaussian i.i.d. input data is often perceived as a strong limitation in the context of statistics and machine learning. In this study, we redeem this line of work in the case of generalized linear classification, a.k.a. the perceptron model, with random labels. We argue that there is a large universality class of high-dimensional input data for which we obtain the same minimum training loss as for Gaussian data with corresponding data covariance. In the limit of vanishing regularization, we further demonstrate that the training loss is independent of the data covariance. On the theoretical side, we prove this universality for an arbitrary mixture of homogeneous Gaussian clouds. Empirically, we show that the universality holds also for a broad range of real datasets. |
| title | Gaussian Universality of Perceptrons with Random Labels |
| topic | Machine Learning Disordered Systems and Neural Networks Probability Statistics Theory |
| url | https://arxiv.org/abs/2205.13303 |