Provable Tempered Overfitting of Minimal Nets and Typical Nets
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916453803884544 |
|---|---|
| author | Harel, Itamar Hoza, William M. Vardi, Gal Evron, Itay Srebro, Nathan Soudry, Daniel |
| author_facet | Harel, Itamar Hoza, William M. Vardi, Gal Evron, Itay Srebro, Nathan Soudry, Daniel |
| contents | We study the overfitting behavior of fully connected deep Neural Networks (NNs) with binary weights fitted to perfectly classify a noisy training set. We consider interpolation using both the smallest NN (having the minimal number of weights) and a random interpolating NN. For both learning rules, we prove overfitting is tempered. Our analysis rests on a new bound on the size of a threshold circuit consistent with a partial function. To the best of our knowledge, ours are the first theoretical results on benign or tempered overfitting that: (1) apply to deep NNs, and (2) do not require a very high or very low input dimension. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19092 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Provable Tempered Overfitting of Minimal Nets and Typical Nets Harel, Itamar Hoza, William M. Vardi, Gal Evron, Itay Srebro, Nathan Soudry, Daniel Machine Learning We study the overfitting behavior of fully connected deep Neural Networks (NNs) with binary weights fitted to perfectly classify a noisy training set. We consider interpolation using both the smallest NN (having the minimal number of weights) and a random interpolating NN. For both learning rules, we prove overfitting is tempered. Our analysis rests on a new bound on the size of a threshold circuit consistent with a partial function. To the best of our knowledge, ours are the first theoretical results on benign or tempered overfitting that: (1) apply to deep NNs, and (2) do not require a very high or very low input dimension. |
| title | Provable Tempered Overfitting of Minimal Nets and Typical Nets |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.19092 |