Large Language Models Assisting Ontology Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910049967800320 |
|---|---|
| author | Lippolis, Anna Sofia Saeedizade, Mohammad Javad Keskisärkkä, Robin Gangemi, Aldo Blomqvist, Eva Nuzzolese, Andrea Giovanni |
| author_facet | Lippolis, Anna Sofia Saeedizade, Mohammad Javad Keskisärkkä, Robin Gangemi, Aldo Blomqvist, Eva Nuzzolese, Andrea Giovanni |
| contents | Ontology evaluation through functional requirements, such as testing via competency question (CQ) verification, is a well-established yet costly, labour-intensive, and error-prone endeavour, even for ontology engineering experts. In this work, we introduce OE-Assist, a novel framework designed to assist ontology evaluation through automated and semi-automated CQ verification. By presenting and leveraging a dataset of 1,393 CQs paired with corresponding ontologies and ontology stories, our contributions present, to our knowledge, the first systematic investigation into large language model (LLM)-assisted ontology evaluation, and include: (i) evaluating the effectiveness of a LLM-based approach for automatically performing CQ verification against a manually created gold standard, and (ii) developing and assessing an LLM-powered framework to assist CQ verification with Protégé, by providing suggestions. We found that automated LLM-based evaluation with o1-preview and o3-mini perform at a similar level to the average user's performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_14552 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Large Language Models Assisting Ontology Evaluation Lippolis, Anna Sofia Saeedizade, Mohammad Javad Keskisärkkä, Robin Gangemi, Aldo Blomqvist, Eva Nuzzolese, Andrea Giovanni Artificial Intelligence Ontology evaluation through functional requirements, such as testing via competency question (CQ) verification, is a well-established yet costly, labour-intensive, and error-prone endeavour, even for ontology engineering experts. In this work, we introduce OE-Assist, a novel framework designed to assist ontology evaluation through automated and semi-automated CQ verification. By presenting and leveraging a dataset of 1,393 CQs paired with corresponding ontologies and ontology stories, our contributions present, to our knowledge, the first systematic investigation into large language model (LLM)-assisted ontology evaluation, and include: (i) evaluating the effectiveness of a LLM-based approach for automatically performing CQ verification against a manually created gold standard, and (ii) developing and assessing an LLM-powered framework to assist CQ verification with Protégé, by providing suggestions. We found that automated LLM-based evaluation with o1-preview and o3-mini perform at a similar level to the average user's performance. |
| title | Large Language Models Assisting Ontology Evaluation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.14552 |