Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.16316 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912324071194624 |
|---|---|
| author | Zhou, Zhanpeng Yang, Yongyi Ren, Jie Sugiyama, Mahito Yan, Junchi |
| author_facet | Zhou, Zhanpeng Yang, Yongyi Ren, Jie Sugiyama, Mahito Yan, Junchi |
| contents | Understanding the learning dynamics of neural networks is a central topic in the deep learning community. In this paper, we take an empirical perspective to study the learning dynamics of neural networks in real-world settings. Specifically, we investigate the evolution process of the empirical Neural Tangent Kernel (eNTK) during training. Our key findings reveal a two-phase learning process: i) in Phase I, the eNTK evolves significantly, signaling the rich regime, and ii) in Phase II, the eNTK keeps evolving but is constrained in a narrow space, a phenomenon we term the cone effect. This two-phase framework builds on the hypothesis proposed by Fort et al. (2020), but we uniquely identify the cone effect in Phase II, demonstrating its significant performance advantages over fully linearized training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16316 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Cone Effect in the Learning Dynamics Zhou, Zhanpeng Yang, Yongyi Ren, Jie Sugiyama, Mahito Yan, Junchi Machine Learning Understanding the learning dynamics of neural networks is a central topic in the deep learning community. In this paper, we take an empirical perspective to study the learning dynamics of neural networks in real-world settings. Specifically, we investigate the evolution process of the empirical Neural Tangent Kernel (eNTK) during training. Our key findings reveal a two-phase learning process: i) in Phase I, the eNTK evolves significantly, signaling the rich regime, and ii) in Phase II, the eNTK keeps evolving but is constrained in a narrow space, a phenomenon we term the cone effect. This two-phase framework builds on the hypothesis proposed by Fort et al. (2020), but we uniquely identify the cone effect in Phase II, demonstrating its significant performance advantages over fully linearized training. |
| title | On the Cone Effect in the Learning Dynamics |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.16316 |