LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916348122103808 |
|---|---|
| author | Kirtania, Shashank Gupta, Priyanshu Radhakirshna, Arjun |
| author_facet | Kirtania, Shashank Gupta, Priyanshu Radhakirshna, Arjun |
| contents | In this paper we examine the limitations of Large Language Models (LLMs) for complex reasoning tasks. Although recent works have started to employ formal languages as an intermediate representation for reasoning tasks, they often face challenges in accurately generating and refining these formal specifications to ensure correctness. To address these issues, this paper proposes Logic-LM++, an improvement on Logic-LM . It uses the ability of LLMs to do pairwise comparisons, allowing the evaluation of the refinements suggested by the LLM. The paper demonstrates that Logic-LM++ outperforms Logic-LM and other contemporary techniques across natural language reasoning tasks on three datasets, FOLIO, ProofWriter and AR-LSAT, with an average improvement of 18.5% on standard prompting, 12.3% on chain of thought prompting and 5% on Logic-LM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02514 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations Kirtania, Shashank Gupta, Priyanshu Radhakirshna, Arjun Logic in Computer Science Artificial Intelligence Computation and Language In this paper we examine the limitations of Large Language Models (LLMs) for complex reasoning tasks. Although recent works have started to employ formal languages as an intermediate representation for reasoning tasks, they often face challenges in accurately generating and refining these formal specifications to ensure correctness. To address these issues, this paper proposes Logic-LM++, an improvement on Logic-LM . It uses the ability of LLMs to do pairwise comparisons, allowing the evaluation of the refinements suggested by the LLM. The paper demonstrates that Logic-LM++ outperforms Logic-LM and other contemporary techniques across natural language reasoning tasks on three datasets, FOLIO, ProofWriter and AR-LSAT, with an average improvement of 18.5% on standard prompting, 12.3% on chain of thought prompting and 5% on Logic-LM. |
| title | LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations |
| topic | Logic in Computer Science Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.02514 |