AmbiGraph-Eval: Can LLMs Effectively Handle Ambiguous Graph Queries?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912535976869888 |
|---|---|
| author | Tian, Yuchen Li, Kaixin Chen, Hao Luo, Ziyang Lin, Hongzhan Schelter, Sebastian Du, Lun Ma, Jing |
| author_facet | Tian, Yuchen Li, Kaixin Chen, Hao Luo, Ziyang Lin, Hongzhan Schelter, Sebastian Du, Lun Ma, Jing |
| contents | Large Language Models (LLMs) have recently demonstrated strong capabilities in translating natural language into database queries, especially when dealing with complex graph-structured data. However, real-world queries often contain inherent ambiguities, and the interconnected nature of graph structures can amplify these challenges, leading to unintended or incorrect query results. To systematically evaluate LLMs on this front, we propose a taxonomy of graph-query ambiguities, comprising three primary types: Attribute Ambiguity, Relationship Ambiguity, and Attribute-Relationship Ambiguity, each subdivided into Same-Entity and Cross-Entity scenarios. We introduce AmbiGraph-Eval, a novel benchmark of real-world ambiguous queries paired with expert-verified graph query answers. Evaluating 9 representative LLMs shows that even top models struggle with ambiguous graph queries. Our findings reveal a critical gap in ambiguity handling and motivate future work on specialized resolution techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09631 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AmbiGraph-Eval: Can LLMs Effectively Handle Ambiguous Graph Queries? Tian, Yuchen Li, Kaixin Chen, Hao Luo, Ziyang Lin, Hongzhan Schelter, Sebastian Du, Lun Ma, Jing Databases Artificial Intelligence Large Language Models (LLMs) have recently demonstrated strong capabilities in translating natural language into database queries, especially when dealing with complex graph-structured data. However, real-world queries often contain inherent ambiguities, and the interconnected nature of graph structures can amplify these challenges, leading to unintended or incorrect query results. To systematically evaluate LLMs on this front, we propose a taxonomy of graph-query ambiguities, comprising three primary types: Attribute Ambiguity, Relationship Ambiguity, and Attribute-Relationship Ambiguity, each subdivided into Same-Entity and Cross-Entity scenarios. We introduce AmbiGraph-Eval, a novel benchmark of real-world ambiguous queries paired with expert-verified graph query answers. Evaluating 9 representative LLMs shows that even top models struggle with ambiguous graph queries. Our findings reveal a critical gap in ambiguity handling and motivate future work on specialized resolution techniques. |
| title | AmbiGraph-Eval: Can LLMs Effectively Handle Ambiguous Graph Queries? |
| topic | Databases Artificial Intelligence |
| url | https://arxiv.org/abs/2508.09631 |