BLAST: Benchmarking LLMs with ASP-based Structured Testing
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915954665979904 |
|---|---|
| author | Santana, Manuel Alejandro Borroto Coppolillo, Erica Calimeri, Francesco Manco, Giuseppe Perri, Simona Ricca, Francesco |
| author_facet | Santana, Manuel Alejandro Borroto Coppolillo, Erica Calimeri, Francesco Manco, Giuseppe Perri, Simona Ricca, Francesco |
| contents | Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dialogue systems, and code generation. Despite evident progress, less attention has been paid to their effectiveness in handling declarative paradigms such as Answer Set Programming (ASP), to date. In this paper we introduce BLAST: The first dedicated benchmarking methodology and associated dataset for evaluating the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework featuring two novel semantic metrics tailored to ASP code generation. The paper presents the results of an empirical evaluation involving ten well-established graph-related problems from the ASP literature and a diverse set of eight state-of-the-art LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_22306 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | BLAST: Benchmarking LLMs with ASP-based Structured Testing Santana, Manuel Alejandro Borroto Coppolillo, Erica Calimeri, Francesco Manco, Giuseppe Perri, Simona Ricca, Francesco Logic in Computer Science Artificial Intelligence Programming Languages Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dialogue systems, and code generation. Despite evident progress, less attention has been paid to their effectiveness in handling declarative paradigms such as Answer Set Programming (ASP), to date. In this paper we introduce BLAST: The first dedicated benchmarking methodology and associated dataset for evaluating the accuracy of LLMs in generating ASP code. BLAST provides a structured evaluation framework featuring two novel semantic metrics tailored to ASP code generation. The paper presents the results of an empirical evaluation involving ten well-established graph-related problems from the ASP literature and a diverse set of eight state-of-the-art LLMs. |
| title | BLAST: Benchmarking LLMs with ASP-based Structured Testing |
| topic | Logic in Computer Science Artificial Intelligence Programming Languages |
| url | https://arxiv.org/abs/2604.22306 |