Logic.py: Bridging the Gap between LLMs and Constraint Solvers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929726048698368 |
|---|---|
| author | Kesseli, Pascal O'Hearn, Peter Cabral, Ricardo Silveira |
| author_facet | Kesseli, Pascal O'Hearn, Peter Cabral, Ricardo Silveira |
| contents | We present a novel approach to formalise and solve search-based problems using large language models, which significantly improves upon previous state-of-the-art results. We demonstrate the efficacy of this approach on the logic puzzles benchmark ZebraLogicBench. Instead of letting the LLM attempt to directly solve the puzzles, our method prompts the model to formalise the problem in a logic-focused domain-specific language (DSL) called Logic.py. This formalised representation is then solved using a constraint solver, leveraging the strengths of both the language model and the solver. Our approach achieves a remarkable 65% absolute improvement over the baseline performance of Llama 3.1 70B on ZebraLogicBench, setting a new state-of-the-art with an accuracy of over 90%. This significant advancement demonstrates the potential of combining language models with domain-specific languages and auxiliary tools on traditionally challenging tasks for LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15776 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Logic.py: Bridging the Gap between LLMs and Constraint Solvers Kesseli, Pascal O'Hearn, Peter Cabral, Ricardo Silveira Artificial Intelligence Logic in Computer Science 68T27 F.4.1; I.2.3; I.2.8 We present a novel approach to formalise and solve search-based problems using large language models, which significantly improves upon previous state-of-the-art results. We demonstrate the efficacy of this approach on the logic puzzles benchmark ZebraLogicBench. Instead of letting the LLM attempt to directly solve the puzzles, our method prompts the model to formalise the problem in a logic-focused domain-specific language (DSL) called Logic.py. This formalised representation is then solved using a constraint solver, leveraging the strengths of both the language model and the solver. Our approach achieves a remarkable 65% absolute improvement over the baseline performance of Llama 3.1 70B on ZebraLogicBench, setting a new state-of-the-art with an accuracy of over 90%. This significant advancement demonstrates the potential of combining language models with domain-specific languages and auxiliary tools on traditionally challenging tasks for LLMs. |
| title | Logic.py: Bridging the Gap between LLMs and Constraint Solvers |
| topic | Artificial Intelligence Logic in Computer Science 68T27 F.4.1; I.2.3; I.2.8 |
| url | https://arxiv.org/abs/2502.15776 |