Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916354771124224 |
|---|---|
| author | Weir, Nathaniel Sanders, Kate Weller, Orion Sharma, Shreya Jiang, Dongwei Jiang, Zhengping Mishra, Bhavana Dalvi Tafjord, Oyvind Jansen, Peter Clark, Peter Van Durme, Benjamin |
| author_facet | Weir, Nathaniel Sanders, Kate Weller, Orion Sharma, Shreya Jiang, Dongwei Jiang, Zhengping Mishra, Bhavana Dalvi Tafjord, Oyvind Jansen, Peter Clark, Peter Van Durme, Benjamin |
| contents | Recent language models enable new opportunities for structured reasoning with text, such as the construction of intuitive, proof-like textual entailment trees without relying on brittle formal logic. However, progress in this direction has been hampered by a long-standing lack of a clear protocol for determining what valid compositional entailment is. This absence causes noisy datasets and limited performance gains by modern neuro-symbolic engines. To address these problems, we formulate a consistent and theoretically grounded approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference. We find that our new dataset, RDTE (Recognizing Decompositional Textual Entailment), has a substantially higher internal consistency (+9%) than prior decompositional entailment datasets. We also find that training an RDTE-oriented entailment classifier via knowledge distillation and employing it in an entailment tree reasoning engine significantly improves both accuracy and proof quality, illustrating the practical benefit of this advance for textual inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14798 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic Weir, Nathaniel Sanders, Kate Weller, Orion Sharma, Shreya Jiang, Dongwei Jiang, Zhengping Mishra, Bhavana Dalvi Tafjord, Oyvind Jansen, Peter Clark, Peter Van Durme, Benjamin Computation and Language Artificial Intelligence Recent language models enable new opportunities for structured reasoning with text, such as the construction of intuitive, proof-like textual entailment trees without relying on brittle formal logic. However, progress in this direction has been hampered by a long-standing lack of a clear protocol for determining what valid compositional entailment is. This absence causes noisy datasets and limited performance gains by modern neuro-symbolic engines. To address these problems, we formulate a consistent and theoretically grounded approach to annotating decompositional entailment and evaluate its impact on LLM-based textual inference. We find that our new dataset, RDTE (Recognizing Decompositional Textual Entailment), has a substantially higher internal consistency (+9%) than prior decompositional entailment datasets. We also find that training an RDTE-oriented entailment classifier via knowledge distillation and employing it in an entailment tree reasoning engine significantly improves both accuracy and proof quality, illustrating the practical benefit of this advance for textual inference. |
| title | Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.14798 |