Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913056730120192 |
|---|---|
| author | Zhang, Zilun Sun, Yutao Zhao, Tiancheng Sha, Leigang Xu, Ruochen Lee, Kyusong Yin, Jianwei |
| author_facet | Zhang, Zilun Sun, Yutao Zhao, Tiancheng Sha, Leigang Xu, Ruochen Lee, Kyusong Yin, Jianwei |
| contents | Humans can retain old knowledge while learning new information, but Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-tuned (SFT) on domain-specific data. Moreover, for Multimodal Large Language Models (MLLMs) which are composed of the LLM base and visual projector (e.g. LLaVA), a significant decline in performance on language benchmarks was observed compared to their single-modality counterparts. To address these challenges, we introduce a novel model-agnostic self-decompression method, Tree Generation (TG), that decompresses knowledge within LLMs into the training corpus. This paper focuses on TG-SFT, which can synthetically generate SFT data for the instruction tuning steps. By incorporating the dumped corpus during SFT for MLLMs, we significantly reduce the forgetting problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11354 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression Zhang, Zilun Sun, Yutao Zhao, Tiancheng Sha, Leigang Xu, Ruochen Lee, Kyusong Yin, Jianwei Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Humans can retain old knowledge while learning new information, but Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-tuned (SFT) on domain-specific data. Moreover, for Multimodal Large Language Models (MLLMs) which are composed of the LLM base and visual projector (e.g. LLaVA), a significant decline in performance on language benchmarks was observed compared to their single-modality counterparts. To address these challenges, we introduce a novel model-agnostic self-decompression method, Tree Generation (TG), that decompresses knowledge within LLMs into the training corpus. This paper focuses on TG-SFT, which can synthetically generate SFT data for the instruction tuning steps. By incorporating the dumped corpus during SFT for MLLMs, we significantly reduce the forgetting problem. |
| title | Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.11354 |