Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.12472 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912712505688064 |
|---|---|
| author | Wang, Mengying Ma, Chenhui Jiao, Ao Liang, Tuo Lu, Pengjun Hegde, Shrinidhi Yin, Yu Gurkan-Cavusoglu, Evren Wu, Yinghui |
| author_facet | Wang, Mengying Ma, Chenhui Jiao, Ao Liang, Tuo Lu, Pengjun Hegde, Shrinidhi Yin, Yu Gurkan-Cavusoglu, Evren Wu, Yinghui |
| contents | Large Language Models (LLMs) have greatly advanced knowledge graph question answering (KGQA), yet existing systems are typically optimized for returning highly relevant but predictable answers. A missing yet desired capacity is to exploit LLMs to suggest surprise and novel ("serendipitious") answers. In this paper, we formally define the serendipity-aware KGQA task and propose the SerenQA framework to evaluate LLMs' ability to uncover unexpected insights in scientific KGQA tasks. SerenQA includes a rigorous serendipity metric based on relevance, novelty, and surprise, along with an expert-annotated benchmark derived from the Clinical Knowledge Graph, focused on drug repurposing. Additionally, it features a structured evaluation pipeline encompassing three subtasks: knowledge retrieval, subgraph reasoning, and serendipity exploration. Our experiments reveal that while state-of-the-art LLMs perform well on retrieval, they still struggle to identify genuinely surprising and valuable discoveries, underscoring a significant room for future improvements. Our curated resources and extended version are released at: https://cwru-db-group.github.io/serenQA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12472 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing Wang, Mengying Ma, Chenhui Jiao, Ao Liang, Tuo Lu, Pengjun Hegde, Shrinidhi Yin, Yu Gurkan-Cavusoglu, Evren Wu, Yinghui Computation and Language Artificial Intelligence Large Language Models (LLMs) have greatly advanced knowledge graph question answering (KGQA), yet existing systems are typically optimized for returning highly relevant but predictable answers. A missing yet desired capacity is to exploit LLMs to suggest surprise and novel ("serendipitious") answers. In this paper, we formally define the serendipity-aware KGQA task and propose the SerenQA framework to evaluate LLMs' ability to uncover unexpected insights in scientific KGQA tasks. SerenQA includes a rigorous serendipity metric based on relevance, novelty, and surprise, along with an expert-annotated benchmark derived from the Clinical Knowledge Graph, focused on drug repurposing. Additionally, it features a structured evaluation pipeline encompassing three subtasks: knowledge retrieval, subgraph reasoning, and serendipity exploration. Our experiments reveal that while state-of-the-art LLMs perform well on retrieval, they still struggle to identify genuinely surprising and valuable discoveries, underscoring a significant room for future improvements. Our curated resources and extended version are released at: https://cwru-db-group.github.io/serenQA. |
| title | Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2511.12472 |