Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.05571 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910906272710656 |
|---|---|
| author | Ovadia, Oded Brief, Meni Lemberg, Rachel Sheetrit, Eitam |
| author_facet | Ovadia, Oded Brief, Meni Lemberg, Rachel Sheetrit, Eitam |
| contents | While Large Language Models (LLMs) acquire vast knowledge during pre-training, they often lack domain-specific, new, or niche information. Continual pre-training (CPT) attempts to address this gap but suffers from catastrophic forgetting and inefficiencies in low-data regimes. We introduce Knowledge-Instruct, a novel approach to efficiently inject knowledge from limited corpora through pure instruction-tuning. By generating information-dense synthetic instruction data, it effectively integrates new knowledge while preserving general reasoning and instruction-following abilities. Knowledge-Instruct demonstrates superior factual memorization, minimizes catastrophic forgetting, and remains scalable by leveraging synthetic data from relatively small language models. Additionally, it enhances contextual understanding, including complex multi-hop reasoning, facilitating integration with retrieval systems. We validate its effectiveness across diverse benchmarks, including Companies, a new dataset that we release to measure knowledge injection capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05571 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions Ovadia, Oded Brief, Meni Lemberg, Rachel Sheetrit, Eitam Computation and Language Artificial Intelligence While Large Language Models (LLMs) acquire vast knowledge during pre-training, they often lack domain-specific, new, or niche information. Continual pre-training (CPT) attempts to address this gap but suffers from catastrophic forgetting and inefficiencies in low-data regimes. We introduce Knowledge-Instruct, a novel approach to efficiently inject knowledge from limited corpora through pure instruction-tuning. By generating information-dense synthetic instruction data, it effectively integrates new knowledge while preserving general reasoning and instruction-following abilities. Knowledge-Instruct demonstrates superior factual memorization, minimizes catastrophic forgetting, and remains scalable by leveraging synthetic data from relatively small language models. Additionally, it enhances contextual understanding, including complex multi-hop reasoning, facilitating integration with retrieval systems. We validate its effectiveness across diverse benchmarks, including Companies, a new dataset that we release to measure knowledge injection capabilities. |
| title | Knowledge-Instruct: Effective Continual Pre-training from Limited Data using Instructions |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.05571 |