Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.22419 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911083634098176 |
|---|---|
| author | Lin, Tung-Wei Fierro, Gabe Li, Han Hong, Tianzhen Nuzzo, Pierluigi Sangiovanni-Vinentelli, Alberto |
| author_facet | Lin, Tung-Wei Fierro, Gabe Li, Han Hong, Tianzhen Nuzzo, Pierluigi Sangiovanni-Vinentelli, Alberto |
| contents | We present a systematic approach for evaluating the quality of knowledge graph repairs with respect to constraint violations defined in shapes constraint language (SHACL). Current evaluation methods rely on \emph{ad hoc} datasets, which limits the rigorous analysis of repair systems in more general settings. Our method addresses this gap by systematically generating violations using a novel mechanism, termed violation-inducing operations (VIOs). We use the proposed evaluation framework to assess a range of repair systems which we build using large language models. We analyze the performance of these systems across different prompting strategies. Results indicate that concise prompts containing both the relevant violated SHACL constraints and key contextual information from the knowledge graph yield the best performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_22419 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Systematic Evaluation of Knowledge Graph Repair with Large Language Models Lin, Tung-Wei Fierro, Gabe Li, Han Hong, Tianzhen Nuzzo, Pierluigi Sangiovanni-Vinentelli, Alberto Databases Artificial Intelligence We present a systematic approach for evaluating the quality of knowledge graph repairs with respect to constraint violations defined in shapes constraint language (SHACL). Current evaluation methods rely on \emph{ad hoc} datasets, which limits the rigorous analysis of repair systems in more general settings. Our method addresses this gap by systematically generating violations using a novel mechanism, termed violation-inducing operations (VIOs). We use the proposed evaluation framework to assess a range of repair systems which we build using large language models. We analyze the performance of these systems across different prompting strategies. Results indicate that concise prompts containing both the relevant violated SHACL constraints and key contextual information from the knowledge graph yield the best performance. |
| title | Systematic Evaluation of Knowledge Graph Repair with Large Language Models |
| topic | Databases Artificial Intelligence |
| url | https://arxiv.org/abs/2507.22419 |