Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913634763931648 |
|---|---|
| author | Fu, Yonggan Yuan, Ye Wu, Shang Yuan, Jiayi Lin, Yingyan Celine |
| author_facet | Fu, Yonggan Yuan, Ye Wu, Shang Yuan, Jiayi Lin, Yingyan Celine |
| contents | Transfer learning leverages feature representations of deep neural networks (DNNs) pretrained on source tasks with rich data to empower effective finetuning on downstream tasks. However, the pretrained models are often prohibitively large for delivering generalizable representations, which limits their deployment on edge devices with constrained resources. To close this gap, we propose a new transfer learning pipeline, which leverages our finding that robust tickets can transfer better, i.e., subnetworks drawn with properly induced adversarial robustness can win better transferability over vanilla lottery ticket subnetworks. Extensive experiments and ablation studies validate that our proposed transfer learning pipeline can achieve enhanced accuracy-sparsity trade-offs across both diverse downstream tasks and sparsity patterns, further enriching the lottery ticket hypothesis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_11834 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning Fu, Yonggan Yuan, Ye Wu, Shang Yuan, Jiayi Lin, Yingyan Celine Machine Learning Transfer learning leverages feature representations of deep neural networks (DNNs) pretrained on source tasks with rich data to empower effective finetuning on downstream tasks. However, the pretrained models are often prohibitively large for delivering generalizable representations, which limits their deployment on edge devices with constrained resources. To close this gap, we propose a new transfer learning pipeline, which leverages our finding that robust tickets can transfer better, i.e., subnetworks drawn with properly induced adversarial robustness can win better transferability over vanilla lottery ticket subnetworks. Extensive experiments and ablation studies validate that our proposed transfer learning pipeline can achieve enhanced accuracy-sparsity trade-offs across both diverse downstream tasks and sparsity patterns, further enriching the lottery ticket hypothesis. |
| title | Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2304.11834 |