FLAME: Empowering Frozen LLMs for Knowledge Graph Completion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908794732150784 |
|---|---|
| author | Xue, Bo Xu, Yi Ma, Bolei Song, Yunchong Ding, Jiaxin Fu, Luoyi Wang, Xinbing |
| author_facet | Xue, Bo Xu, Yi Ma, Bolei Song, Yunchong Ding, Jiaxin Fu, Luoyi Wang, Xinbing |
| contents | Traditional knowledge graph completion (KGC) methods rely solely on structural information and struggle with sparsity, while Large Language Models (LLMs) address these limitations through rich world knowledge and strong context modeling. Fine-tuning LLMs is effective but costly, while non-fine-tuned LLMs are efficient but suboptimal. To address this trade-off, we propose \textbf{FLAME}, a framework that extracts context-aware hidden states from intermediate layers of frozen LLMs to train data-efficient KGC classifiers. We bridge LLM-KG semantic gaps via subgraph-based entity descriptions and employ sliced mutual information (SMI) to quantify task-relevant information in representations. Experiments demonstrate that FLAME achieves 47\% improvement over non-fine-tuned LLM baselines and, to our knowledge, is the first to achieve fine-tuned performance with $188\times$ memory efficiency and $26.11\times$ speedup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_06787 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FLAME: Empowering Frozen LLMs for Knowledge Graph Completion Xue, Bo Xu, Yi Ma, Bolei Song, Yunchong Ding, Jiaxin Fu, Luoyi Wang, Xinbing Computation and Language Traditional knowledge graph completion (KGC) methods rely solely on structural information and struggle with sparsity, while Large Language Models (LLMs) address these limitations through rich world knowledge and strong context modeling. Fine-tuning LLMs is effective but costly, while non-fine-tuned LLMs are efficient but suboptimal. To address this trade-off, we propose \textbf{FLAME}, a framework that extracts context-aware hidden states from intermediate layers of frozen LLMs to train data-efficient KGC classifiers. We bridge LLM-KG semantic gaps via subgraph-based entity descriptions and employ sliced mutual information (SMI) to quantify task-relevant information in representations. Experiments demonstrate that FLAME achieves 47\% improvement over non-fine-tuned LLM baselines and, to our knowledge, is the first to achieve fine-tuned performance with $188\times$ memory efficiency and $26.11\times$ speedup. |
| title | FLAME: Empowering Frozen LLMs for Knowledge Graph Completion |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2408.06787 |