Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929532550774784 |
|---|---|
| author | Mannelli, Stefano Sarao Ivashynka, Yaraslau Saxe, Andrew Saglietti, Luca |
| author_facet | Mannelli, Stefano Sarao Ivashynka, Yaraslau Saxe, Andrew Saglietti, Luca |
| contents | A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks. According to the lottery ticket hypothesis, overparameterised networks have an increased chance of containing a sub-network that is well-initialised to solve the task at hand. A more parsimonious approach, inspired by animal learning, consists in guiding the learner towards solving the task by curating the order of the examples, i.e. providing a curriculum. However, this learning strategy seems to be hardly beneficial in deep learning applications. In this work, we undertake an analytical study that connects curriculum learning and overparameterisation. In particular, we investigate their interplay in the online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results show that a high degree of overparameterisation -- while simplifying the problem -- can limit the benefit from curricula, providing a theoretical account of the ineffectiveness of curricula in deep learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_01589 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks Mannelli, Stefano Sarao Ivashynka, Yaraslau Saxe, Andrew Saglietti, Luca Machine Learning Disordered Systems and Neural Networks Neurons and Cognition A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks. According to the lottery ticket hypothesis, overparameterised networks have an increased chance of containing a sub-network that is well-initialised to solve the task at hand. A more parsimonious approach, inspired by animal learning, consists in guiding the learner towards solving the task by curating the order of the examples, i.e. providing a curriculum. However, this learning strategy seems to be hardly beneficial in deep learning applications. In this work, we undertake an analytical study that connects curriculum learning and overparameterisation. In particular, we investigate their interplay in the online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results show that a high degree of overparameterisation -- while simplifying the problem -- can limit the benefit from curricula, providing a theoretical account of the ineffectiveness of curricula in deep learning. |
| title | Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks |
| topic | Machine Learning Disordered Systems and Neural Networks Neurons and Cognition |
| url | https://arxiv.org/abs/2406.01589 |