A Second-Order Perspective on Model Compositionality and Incremental Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910851242393600 |
|---|---|
| author | Porrello, Angelo Bonicelli, Lorenzo Buzzega, Pietro Millunzi, Monica Calderara, Simone Cucchiara, Rita |
| author_facet | Porrello, Angelo Bonicelli, Lorenzo Buzzega, Pietro Millunzi, Monica Calderara, Simone Cucchiara, Rita |
| contents | The fine-tuning of deep pre-trained models has revealed compositional properties, with multiple specialized modules that can be arbitrarily composed into a single, multi-task model. However, identifying the conditions that promote compositionality remains an open issue, with recent efforts concentrating mainly on linearized networks. We conduct a theoretical study that attempts to demystify compositionality in standard non-linear networks through the second-order Taylor approximation of the loss function. The proposed formulation highlights the importance of staying within the pre-training basin to achieve composable modules. Moreover, it provides the basis for two dual incremental training algorithms: the one from the perspective of multiple models trained individually, while the other aims to optimize the composed model as a whole. We probe their application in incremental classification tasks and highlight some valuable skills. In fact, the pool of incrementally learned modules not only supports the creation of an effective multi-task model but also enables unlearning and specialization in certain tasks. Code available at https://github.com/aimagelab/mammoth. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_16350 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Second-Order Perspective on Model Compositionality and Incremental Learning Porrello, Angelo Bonicelli, Lorenzo Buzzega, Pietro Millunzi, Monica Calderara, Simone Cucchiara, Rita Artificial Intelligence Machine Learning The fine-tuning of deep pre-trained models has revealed compositional properties, with multiple specialized modules that can be arbitrarily composed into a single, multi-task model. However, identifying the conditions that promote compositionality remains an open issue, with recent efforts concentrating mainly on linearized networks. We conduct a theoretical study that attempts to demystify compositionality in standard non-linear networks through the second-order Taylor approximation of the loss function. The proposed formulation highlights the importance of staying within the pre-training basin to achieve composable modules. Moreover, it provides the basis for two dual incremental training algorithms: the one from the perspective of multiple models trained individually, while the other aims to optimize the composed model as a whole. We probe their application in incremental classification tasks and highlight some valuable skills. In fact, the pool of incrementally learned modules not only supports the creation of an effective multi-task model but also enables unlearning and specialization in certain tasks. Code available at https://github.com/aimagelab/mammoth. |
| title | A Second-Order Perspective on Model Compositionality and Incremental Learning |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2405.16350 |