Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.12545 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917886956666880 |
|---|---|
| author | Wang, Hui Yip, Cho Tung Li, Bo |
| author_facet | Wang, Hui Yip, Cho Tung Li, Bo |
| contents | Gradient-based meta-learning algorithms have gained popularity for their ability to train models on new tasks using limited data. Empirical observations indicate that such algorithms are able to learn a shared representation across tasks, which is regarded as a key factor in their success. However, the in-depth theoretical understanding of the learning dynamics and the origin of the shared representation remains underdeveloped. In this work, we investigate the meta-learning dynamics of nonlinear two-layer neural networks trained on streaming tasks in the teacher-student scenario. Through the lens of statistical physics analysis, we characterize the macroscopic behavior of the meta-training processes, the formation of the shared representation, and the generalization ability of the model on new tasks. The analysis also points to the importance of the choice of certain hyperparameters of the learning algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12545 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamics of Meta-learning Representation in the Teacher-student Scenario Wang, Hui Yip, Cho Tung Li, Bo Machine Learning Disordered Systems and Neural Networks Gradient-based meta-learning algorithms have gained popularity for their ability to train models on new tasks using limited data. Empirical observations indicate that such algorithms are able to learn a shared representation across tasks, which is regarded as a key factor in their success. However, the in-depth theoretical understanding of the learning dynamics and the origin of the shared representation remains underdeveloped. In this work, we investigate the meta-learning dynamics of nonlinear two-layer neural networks trained on streaming tasks in the teacher-student scenario. Through the lens of statistical physics analysis, we characterize the macroscopic behavior of the meta-training processes, the formation of the shared representation, and the generalization ability of the model on new tasks. The analysis also points to the importance of the choice of certain hyperparameters of the learning algorithms. |
| title | Dynamics of Meta-learning Representation in the Teacher-student Scenario |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2408.12545 |