Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910036034322432 |
|---|---|
| author | Choubey, Prafulla Kumar Su, Xin Luo, Man Peng, Xiangyu Xiong, Caiming Le, Tiep Rosenman, Shachar Lal, Vasudev Mui, Phil Ho, Ricky Howard, Phillip Wu, Chien-Sheng |
| author_facet | Choubey, Prafulla Kumar Su, Xin Luo, Man Peng, Xiangyu Xiong, Caiming Le, Tiep Rosenman, Shachar Lal, Vasudev Mui, Phil Ho, Ricky Howard, Phillip Wu, Chien-Sheng |
| contents | Document-level knowledge graph (KG) construction faces a fundamental scaling challenge: existing methods either rely on expensive large language models (LLMs), making them economically nonviable for large-scale corpora, or employ smaller models that produce incomplete and inconsistent graphs. We find that this limitation stems not from model capabilities but from insufficient training on high-quality document-level KG data. To address this gap, we introduce SynthKG, a multi-step data synthesis pipeline that generates high-quality document-KG pairs through systematic chunking, decontextualization, and structured extraction using LLMs. By fine-tuning a smaller LLM on synthesized document-KG pairs, we streamline the multi-step process into a single-step KG generation approach called Distill-SynthKG. Furthermore, we repurpose existing question-answering datasets to construct KG evaluation datasets and introduce new evaluation metrics. Using KGs produced by Distill-SynthKG, we also design a novel graph-based retrieval framework for RAG. Experimental results demonstrate that Distill-SynthKG not only surpasses all baseline models in KG quality (including models up to eight times larger) but also consistently improves in retrieval and question-answering tasks. Additionally, our proposed graph retrieval framework outperforms all KG-retrieval methods across multiple benchmark datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16597 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation Choubey, Prafulla Kumar Su, Xin Luo, Man Peng, Xiangyu Xiong, Caiming Le, Tiep Rosenman, Shachar Lal, Vasudev Mui, Phil Ho, Ricky Howard, Phillip Wu, Chien-Sheng Computation and Language Information Retrieval Document-level knowledge graph (KG) construction faces a fundamental scaling challenge: existing methods either rely on expensive large language models (LLMs), making them economically nonviable for large-scale corpora, or employ smaller models that produce incomplete and inconsistent graphs. We find that this limitation stems not from model capabilities but from insufficient training on high-quality document-level KG data. To address this gap, we introduce SynthKG, a multi-step data synthesis pipeline that generates high-quality document-KG pairs through systematic chunking, decontextualization, and structured extraction using LLMs. By fine-tuning a smaller LLM on synthesized document-KG pairs, we streamline the multi-step process into a single-step KG generation approach called Distill-SynthKG. Furthermore, we repurpose existing question-answering datasets to construct KG evaluation datasets and introduce new evaluation metrics. Using KGs produced by Distill-SynthKG, we also design a novel graph-based retrieval framework for RAG. Experimental results demonstrate that Distill-SynthKG not only surpasses all baseline models in KG quality (including models up to eight times larger) but also consistently improves in retrieval and question-answering tasks. Additionally, our proposed graph retrieval framework outperforms all KG-retrieval methods across multiple benchmark datasets. |
| title | Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2410.16597 |