Rehearsal-Free Modular and Compositional Continual Learning for Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917627100659712 |
|---|---|
| author | Wang, Mingyang Adel, Heike Lange, Lukas Strötgen, Jannik Schütze, Hinrich |
| author_facet | Wang, Mingyang Adel, Heike Lange, Lukas Strötgen, Jannik Schütze, Hinrich |
| contents | Continual learning aims at incrementally acquiring new knowledge while not forgetting existing knowledge. To overcome catastrophic forgetting, methods are either rehearsal-based, i.e., store data examples from previous tasks for data replay, or isolate parameters dedicated to each task. However, rehearsal-based methods raise privacy and memory issues, and parameter-isolation continual learning does not consider interaction between tasks, thus hindering knowledge transfer. In this work, we propose MoCL, a rehearsal-free Modular and Compositional Continual Learning framework which continually adds new modules to language models and composes them with existing modules. Experiments on various benchmarks show that MoCL outperforms state of the art and effectively facilitates knowledge transfer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_00790 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rehearsal-Free Modular and Compositional Continual Learning for Language Models Wang, Mingyang Adel, Heike Lange, Lukas Strötgen, Jannik Schütze, Hinrich Machine Learning Computation and Language Continual learning aims at incrementally acquiring new knowledge while not forgetting existing knowledge. To overcome catastrophic forgetting, methods are either rehearsal-based, i.e., store data examples from previous tasks for data replay, or isolate parameters dedicated to each task. However, rehearsal-based methods raise privacy and memory issues, and parameter-isolation continual learning does not consider interaction between tasks, thus hindering knowledge transfer. In this work, we propose MoCL, a rehearsal-free Modular and Compositional Continual Learning framework which continually adds new modules to language models and composes them with existing modules. Experiments on various benchmarks show that MoCL outperforms state of the art and effectively facilitates knowledge transfer. |
| title | Rehearsal-Free Modular and Compositional Continual Learning for Language Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2404.00790 |