Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.17479 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916262680985600 |
|---|---|
| author | Zhou, Zhangchen Zhang, Yaoyu Xu, Zhi-Qin John |
| author_facet | Zhou, Zhangchen Zhang, Yaoyu Xu, Zhi-Qin John |
| contents | Grokking is the phenomenon where neural networks NNs initially fit the training data and later generalize to the test data during training. In this paper, we empirically provide a frequency perspective to explain the emergence of this phenomenon in NNs. The core insight is that the networks initially learn the less salient frequency components present in the test data. We observe this phenomenon across both synthetic and real datasets, offering a novel viewpoint for elucidating the grokking phenomenon by characterizing it through the lens of frequency dynamics during the training process. Our empirical frequency-based analysis sheds new light on understanding the grokking phenomenon and its underlying mechanisms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17479 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A rationale from frequency perspective for grokking in training neural network Zhou, Zhangchen Zhang, Yaoyu Xu, Zhi-Qin John Machine Learning Neural and Evolutionary Computing Grokking is the phenomenon where neural networks NNs initially fit the training data and later generalize to the test data during training. In this paper, we empirically provide a frequency perspective to explain the emergence of this phenomenon in NNs. The core insight is that the networks initially learn the less salient frequency components present in the test data. We observe this phenomenon across both synthetic and real datasets, offering a novel viewpoint for elucidating the grokking phenomenon by characterizing it through the lens of frequency dynamics during the training process. Our empirical frequency-based analysis sheds new light on understanding the grokking phenomenon and its underlying mechanisms. |
| title | A rationale from frequency perspective for grokking in training neural network |
| topic | Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2405.17479 |