Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.19316 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909020735930368 |
|---|---|
| author | Jiang, Kailin Jiang, Hongbo Jiang, Ning Gao, Zhi Bi, Jinhe Ren, Yuchen Li, Bin Du, Yuntao Liu, Lei Li, Qing |
| author_facet | Jiang, Kailin Jiang, Hongbo Jiang, Ning Gao, Zhi Bi, Jinhe Ren, Yuchen Li, Bin Du, Yuntao Liu, Lei Li, Qing |
| contents | Large Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowledge acquisition. Effective knowledge injection thus becomes critical, involving two goals: knowledge adaptation (injecting new knowledge) and knowledge retention (preserving old knowledge). Existing methods often struggle to learn new knowledge and suffer from catastrophic forgetting. To address this, we propose KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints for injecting new knowledge into large multimodal models while preserving old knowledge. Unlike general text or image data augmentation, KORE automatically converts individual knowledge items into structured and comprehensive knowledge to ensure that the model accurately learns new knowledge, enabling accurate adaptation. Meanwhile, KORE stores previous knowledge in the covariance matrix of LMM's linear layer activations and initializes the adapter by projecting the original weights into the matrix's null space, defining a fine-tuning direction that minimizes interference with previous knowledge, enabling powerful retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B, LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new knowledge injection performance and effectively mitigates catastrophic forgetting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19316 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls Jiang, Kailin Jiang, Hongbo Jiang, Ning Gao, Zhi Bi, Jinhe Ren, Yuchen Li, Bin Du, Yuntao Liu, Lei Li, Qing Computation and Language Large Multimodal Models encode extensive factual knowledge in their pre-trained weights. However, its knowledge remains static and limited, unable to keep pace with real-world developments, which hinders continuous knowledge acquisition. Effective knowledge injection thus becomes critical, involving two goals: knowledge adaptation (injecting new knowledge) and knowledge retention (preserving old knowledge). Existing methods often struggle to learn new knowledge and suffer from catastrophic forgetting. To address this, we propose KORE, a synergistic method of KnOwledge-oRientEd augmentations and constraints for injecting new knowledge into large multimodal models while preserving old knowledge. Unlike general text or image data augmentation, KORE automatically converts individual knowledge items into structured and comprehensive knowledge to ensure that the model accurately learns new knowledge, enabling accurate adaptation. Meanwhile, KORE stores previous knowledge in the covariance matrix of LMM's linear layer activations and initializes the adapter by projecting the original weights into the matrix's null space, defining a fine-tuning direction that minimizes interference with previous knowledge, enabling powerful retention. Extensive experiments on various LMMs, including LLaVA-v1.5-7B, LLaVA-v1.5-13B, and Qwen2.5-VL-7B, show that KORE achieves superior new knowledge injection performance and effectively mitigates catastrophic forgetting. |
| title | KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.19316 |