Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916348272050176 |
|---|---|
| author | Steinigen, Daniel Teucher, Roman Ruland, Timm Heine Rudat, Max Flores-Herr, Nicolas Fischer, Peter Milosevic, Nikola Schymura, Christopher Ziletti, Angelo |
| author_facet | Steinigen, Daniel Teucher, Roman Ruland, Timm Heine Rudat, Max Flores-Herr, Nicolas Fischer, Peter Milosevic, Nikola Schymura, Christopher Ziletti, Angelo |
| contents | Recent advancements in Large Language Models (LLMs) have showcased their proficiency in answering natural language queries. However, their effectiveness is hindered by limited domain-specific knowledge, raising concerns about the reliability of their responses. We introduce a hybrid system that augments LLMs with domain-specific knowledge graphs (KGs), thereby aiming to enhance factual correctness using a KG-based retrieval approach. We focus on a medical KG to demonstrate our methodology, which includes (1) pre-processing, (2) Cypher query generation, (3) Cypher query processing, (4) KG retrieval, and (5) LLM-enhanced response generation. We evaluate our system on a curated dataset of 69 samples, achieving a precision of 78\% in retrieving correct KG nodes. Our findings indicate that the hybrid system surpasses a standalone LLM in accuracy and completeness, as verified by an LLM-as-a-Judge evaluation method. This positions the system as a promising tool for applications that demand factual correctness and completeness, such as target identification -- a critical process in pinpointing biological entities for disease treatment or crop enhancement. Moreover, its intuitive search interface and ability to provide accurate responses within seconds make it well-suited for time-sensitive, precision-focused research contexts. We publish the source code together with the dataset and the prompt templates used. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03010 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs Steinigen, Daniel Teucher, Roman Ruland, Timm Heine Rudat, Max Flores-Herr, Nicolas Fischer, Peter Milosevic, Nikola Schymura, Christopher Ziletti, Angelo Computation and Language Information Retrieval Recent advancements in Large Language Models (LLMs) have showcased their proficiency in answering natural language queries. However, their effectiveness is hindered by limited domain-specific knowledge, raising concerns about the reliability of their responses. We introduce a hybrid system that augments LLMs with domain-specific knowledge graphs (KGs), thereby aiming to enhance factual correctness using a KG-based retrieval approach. We focus on a medical KG to demonstrate our methodology, which includes (1) pre-processing, (2) Cypher query generation, (3) Cypher query processing, (4) KG retrieval, and (5) LLM-enhanced response generation. We evaluate our system on a curated dataset of 69 samples, achieving a precision of 78\% in retrieving correct KG nodes. Our findings indicate that the hybrid system surpasses a standalone LLM in accuracy and completeness, as verified by an LLM-as-a-Judge evaluation method. This positions the system as a promising tool for applications that demand factual correctness and completeness, such as target identification -- a critical process in pinpointing biological entities for disease treatment or crop enhancement. Moreover, its intuitive search interface and ability to provide accurate responses within seconds make it well-suited for time-sensitive, precision-focused research contexts. We publish the source code together with the dataset and the prompt templates used. |
| title | Fact Finder -- Enhancing Domain Expertise of Large Language Models by Incorporating Knowledge Graphs |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2408.03010 |