Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908860697018368 |
|---|---|
| author | Westerhoff, Justus Atefi, Golzar Koddenbrock, Mario Figueroa, Alexei Löser, Alexander Rodner, Erik Gers, Felix A. |
| author_facet | Westerhoff, Justus Atefi, Golzar Koddenbrock, Mario Figueroa, Alexei Löser, Alexander Rodner, Erik Gers, Felix A. |
| contents | The capacity of foundation models allows for their application to new, unseen tasks. The adaptation to such tasks is called transfer learning. An efficient transfer learning method that circumvents parameter optimization is imprinting. The conceptual differences between studies on imprinting form the basis of our systematic investigation. In this work, we propose the general \texttt{IMPRINT} framework, identifying three main components: generation, normalization, and aggregation. Through the lens of this framework, we conduct an in-depth analysis and a comparison of the existing methods. Our findings reveal the benefits of representing novel data with multiple proxies in the generation step and show the importance of proper normalization. Beyond an extensive analytical grounding, our framework enables us to propose a novel variant of imprinting which outperforms previous work on transfer learning tasks by 4\%. This variant determines proxies through clustering motivated by the neural collapse phenomenon -- a connection that we draw for the first time. We publicly release our code at https://github.com/DATEXIS/IMPRINT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14572 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation Westerhoff, Justus Atefi, Golzar Koddenbrock, Mario Figueroa, Alexei Löser, Alexander Rodner, Erik Gers, Felix A. Machine Learning Artificial Intelligence The capacity of foundation models allows for their application to new, unseen tasks. The adaptation to such tasks is called transfer learning. An efficient transfer learning method that circumvents parameter optimization is imprinting. The conceptual differences between studies on imprinting form the basis of our systematic investigation. In this work, we propose the general \texttt{IMPRINT} framework, identifying three main components: generation, normalization, and aggregation. Through the lens of this framework, we conduct an in-depth analysis and a comparison of the existing methods. Our findings reveal the benefits of representing novel data with multiple proxies in the generation step and show the importance of proper normalization. Beyond an extensive analytical grounding, our framework enables us to propose a novel variant of imprinting which outperforms previous work on transfer learning tasks by 4\%. This variant determines proxies through clustering motivated by the neural collapse phenomenon -- a connection that we draw for the first time. We publicly release our code at https://github.com/DATEXIS/IMPRINT. |
| title | Robust Weight Imprinting: Insights from Neural Collapse and Proxy-Based Aggregation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.14572 |