IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914109100916736 |
|---|---|
| author | Zhang, Tianyi Mai, Florian Flek, Lucie |
| author_facet | Zhang, Tianyi Mai, Florian Flek, Lucie |
| contents | Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20377 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation Zhang, Tianyi Mai, Florian Flek, Lucie Artificial Intelligence Computation and Language Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to instruction-tuned models is known to degrade their instruction-following capability and semantic representations. Existing fixes assume access to the original base model or rely on knowledge from an external domain-specific database - both of which pose a realistic barrier in settings where the base model weights are withheld for safety reasons or reliable external corpora are unavailable. In this work, we propose Instruction-Knowledge-Aware Continual Adaptation (IKnow), a simple and general framework that formulates novel self-supervised objectives in the instruction-response dialogue format. Rather than depend- ing on external resources, IKnow leverages domain knowledge embedded within the text itself and learns to encode it at a deeper semantic level. |
| title | IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.20377 |