Study of Training Dynamics for Memory-Constrained Fine-Tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908841896050688 |
|---|---|
| author | Quélennec, Aël Hezbri, Nour Mozharovskyi, Pavlo Nguyen, Van-Tam Tartaglione, Enzo |
| author_facet | Quélennec, Aël Hezbri, Nour Mozharovskyi, Pavlo Nguyen, Van-Tam Tartaglione, Enzo |
| contents | Memory-efficient training of deep neural networks has become increasingly important as models grow larger while deployment environments impose strict resource constraints. We propose TraDy, a novel transfer learning scheme leveraging two key insights: layer importance for updates is architecture-dependent and determinable a priori, while dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. We introduce a dynamic channel selection approach that stochastically resamples channels between epochs within preselected layers. Extensive experiments demonstrate TraDy achieves state-of-the-art performance across various downstream tasks and architectures while maintaining strict memory constraints, achieving up to 99% activation sparsity, 95% weight derivative sparsity, and 97% reduction in FLOPs for weight derivative computation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19675 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Study of Training Dynamics for Memory-Constrained Fine-Tuning Quélennec, Aël Hezbri, Nour Mozharovskyi, Pavlo Nguyen, Van-Tam Tartaglione, Enzo Machine Learning Artificial Intelligence Memory-efficient training of deep neural networks has become increasingly important as models grow larger while deployment environments impose strict resource constraints. We propose TraDy, a novel transfer learning scheme leveraging two key insights: layer importance for updates is architecture-dependent and determinable a priori, while dynamic stochastic channel selection provides superior gradient approximation compared to static approaches. We introduce a dynamic channel selection approach that stochastically resamples channels between epochs within preselected layers. Extensive experiments demonstrate TraDy achieves state-of-the-art performance across various downstream tasks and architectures while maintaining strict memory constraints, achieving up to 99% activation sparsity, 95% weight derivative sparsity, and 97% reduction in FLOPs for weight derivative computation. |
| title | Study of Training Dynamics for Memory-Constrained Fine-Tuning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.19675 |