Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911822693531648 |
|---|---|
| author | Lyu, Kaifeng Jin, Jikai Li, Zhiyuan Du, Simon S. Lee, Jason D. Hu, Wei |
| author_facet | Lyu, Kaifeng Jin, Jikai Li, Zhiyuan Du, Simon S. Lee, Jason D. Hu, Wei |
| contents | Recent work by Power et al. (2022) highlighted a surprising "grokking" phenomenon in learning arithmetic tasks: a neural net first "memorizes" the training set, resulting in perfect training accuracy but near-random test accuracy, and after training for sufficiently longer, it suddenly transitions to perfect test accuracy. This paper studies the grokking phenomenon in theoretical setups and shows that it can be induced by a dichotomy of early and late phase implicit biases. Specifically, when training homogeneous neural nets with large initialization and small weight decay on both classification and regression tasks, we prove that the training process gets trapped at a solution corresponding to a kernel predictor for a long time, and then a very sharp transition to min-norm/max-margin predictors occurs, leading to a dramatic change in test accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_18817 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking Lyu, Kaifeng Jin, Jikai Li, Zhiyuan Du, Simon S. Lee, Jason D. Hu, Wei Machine Learning Artificial Intelligence Recent work by Power et al. (2022) highlighted a surprising "grokking" phenomenon in learning arithmetic tasks: a neural net first "memorizes" the training set, resulting in perfect training accuracy but near-random test accuracy, and after training for sufficiently longer, it suddenly transitions to perfect test accuracy. This paper studies the grokking phenomenon in theoretical setups and shows that it can be induced by a dichotomy of early and late phase implicit biases. Specifically, when training homogeneous neural nets with large initialization and small weight decay on both classification and regression tasks, we prove that the training process gets trapped at a solution corresponding to a kernel predictor for a long time, and then a very sharp transition to min-norm/max-margin predictors occurs, leading to a dramatic change in test accuracy. |
| title | Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2311.18817 |