KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909241352126464 |
|---|---|
| author | Yang, Rui Liu, Haoran Marrese-Taylor, Edison Zeng, Qingcheng Ke, Yu He Li, Wanxin Cheng, Lechao Chen, Qingyu Caverlee, James Matsuo, Yutaka Li, Irene |
| author_facet | Yang, Rui Liu, Haoran Marrese-Taylor, Edison Zeng, Qingcheng Ke, Yu He Li, Wanxin Cheng, Lechao Chen, Qingyu Caverlee, James Matsuo, Yutaka Li, Irene |
| contents | Large language models (LLMs) have demonstrated impressive generative capabilities with the potential to innovate in medicine. However, the application of LLMs in real clinical settings remains challenging due to the lack of factual consistency in the generated content. In this work, we develop an augmented LLM framework, KG-Rank, which leverages a medical knowledge graph (KG) along with ranking and re-ranking techniques, to improve the factuality of long-form question answering (QA) in the medical domain. Specifically, when receiving a question, KG-Rank automatically identifies medical entities within the question and retrieves the related triples from the medical KG to gather factual information. Subsequently, KG-Rank innovatively applies multiple ranking techniques to refine the ordering of these triples, providing more relevant and precise information for LLM inference. To the best of our knowledge, KG-Rank is the first application of KG combined with ranking models in medical QA specifically for generating long answers. Evaluation on four selected medical QA datasets demonstrates that KG-Rank achieves an improvement of over 18% in ROUGE-L score. Additionally, we extend KG-Rank to open domains, including law, business, music, and history, where it realizes a 14% improvement in ROUGE-L score, indicating the effectiveness and great potential of KG-Rank. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05881 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques Yang, Rui Liu, Haoran Marrese-Taylor, Edison Zeng, Qingcheng Ke, Yu He Li, Wanxin Cheng, Lechao Chen, Qingyu Caverlee, James Matsuo, Yutaka Li, Irene Computation and Language Large language models (LLMs) have demonstrated impressive generative capabilities with the potential to innovate in medicine. However, the application of LLMs in real clinical settings remains challenging due to the lack of factual consistency in the generated content. In this work, we develop an augmented LLM framework, KG-Rank, which leverages a medical knowledge graph (KG) along with ranking and re-ranking techniques, to improve the factuality of long-form question answering (QA) in the medical domain. Specifically, when receiving a question, KG-Rank automatically identifies medical entities within the question and retrieves the related triples from the medical KG to gather factual information. Subsequently, KG-Rank innovatively applies multiple ranking techniques to refine the ordering of these triples, providing more relevant and precise information for LLM inference. To the best of our knowledge, KG-Rank is the first application of KG combined with ranking models in medical QA specifically for generating long answers. Evaluation on four selected medical QA datasets demonstrates that KG-Rank achieves an improvement of over 18% in ROUGE-L score. Additionally, we extend KG-Rank to open domains, including law, business, music, and history, where it realizes a 14% improvement in ROUGE-L score, indicating the effectiveness and great potential of KG-Rank. |
| title | KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.05881 |