Preserving Generalization of Language models in Few-shot Continual Relation Extraction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929522399510528 |
|---|---|
| author | Tran, Quyen Thanh, Nguyen Xuan Anh, Nguyen Hoang Hai, Nam Le Le, Trung Van Ngo, Linh Nguyen, Thien Huu |
| author_facet | Tran, Quyen Thanh, Nguyen Xuan Anh, Nguyen Hoang Hai, Nam Le Le, Trung Van Ngo, Linh Nguyen, Thien Huu |
| contents | Few-shot Continual Relations Extraction (FCRE) is an emerging and dynamic area of study where models can sequentially integrate knowledge from new relations with limited labeled data while circumventing catastrophic forgetting and preserving prior knowledge from pre-trained backbones. In this work, we introduce a novel method that leverages often-discarded language model heads. By employing these components via a mutual information maximization strategy, our approach helps maintain prior knowledge from the pre-trained backbone and strategically aligns the primary classification head, thereby enhancing model performance. Furthermore, we explore the potential of Large Language Models (LLMs), renowned for their wealth of knowledge, in addressing FCRE challenges. Our comprehensive experimental results underscore the efficacy of the proposed method and offer valuable insights for future work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00334 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preserving Generalization of Language models in Few-shot Continual Relation Extraction Tran, Quyen Thanh, Nguyen Xuan Anh, Nguyen Hoang Hai, Nam Le Le, Trung Van Ngo, Linh Nguyen, Thien Huu Computation and Language Artificial Intelligence Few-shot Continual Relations Extraction (FCRE) is an emerging and dynamic area of study where models can sequentially integrate knowledge from new relations with limited labeled data while circumventing catastrophic forgetting and preserving prior knowledge from pre-trained backbones. In this work, we introduce a novel method that leverages often-discarded language model heads. By employing these components via a mutual information maximization strategy, our approach helps maintain prior knowledge from the pre-trained backbone and strategically aligns the primary classification head, thereby enhancing model performance. Furthermore, we explore the potential of Large Language Models (LLMs), renowned for their wealth of knowledge, in addressing FCRE challenges. Our comprehensive experimental results underscore the efficacy of the proposed method and offer valuable insights for future work. |
| title | Preserving Generalization of Language models in Few-shot Continual Relation Extraction |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.00334 |