Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909230918795264 |
|---|---|
| author | Saleem, Tausifa Jan Ahuja, Ramanjit Prasad, Surendra Lall, Brejesh |
| author_facet | Saleem, Tausifa Jan Ahuja, Ramanjit Prasad, Surendra Lall, Brejesh |
| contents | Lottery ticket hypothesis for deep neural networks emphasizes the importance of initialization used to re-train the sparser networks obtained using the iterative magnitude pruning process. An explanation for why the specific initialization proposed by the lottery ticket hypothesis tends to work better in terms of generalization (and training) performance has been lacking. Moreover, the underlying principles in iterative magnitude pruning, like the pruning of smaller magnitude weights and the role of the iterative process, lack full understanding and explanation. In this work, we attempt to provide insights into these phenomena by empirically studying the volume/geometry and loss landscape characteristics of the solutions obtained at various stages of the iterative magnitude pruning process. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_15022 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning Saleem, Tausifa Jan Ahuja, Ramanjit Prasad, Surendra Lall, Brejesh Machine Learning Lottery ticket hypothesis for deep neural networks emphasizes the importance of initialization used to re-train the sparser networks obtained using the iterative magnitude pruning process. An explanation for why the specific initialization proposed by the lottery ticket hypothesis tends to work better in terms of generalization (and training) performance has been lacking. Moreover, the underlying principles in iterative magnitude pruning, like the pruning of smaller magnitude weights and the role of the iterative process, lack full understanding and explanation. In this work, we attempt to provide insights into these phenomena by empirically studying the volume/geometry and loss landscape characteristics of the solutions obtained at various stages of the iterative magnitude pruning process. |
| title | Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2403.15022 |