Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.11335 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912194581495808 |
|---|---|
| author | Erwin, Kyle Axelrod, Guy Chang, Maria Fokoue, Achille Crouse, Maxwell Dan, Soham Gao, Tian Uceda-Sosa, Rosario Makondo, Ndivhuwo Khan, Naweed Gray, Alexander |
| author_facet | Erwin, Kyle Axelrod, Guy Chang, Maria Fokoue, Achille Crouse, Maxwell Dan, Soham Gao, Tian Uceda-Sosa, Rosario Makondo, Ndivhuwo Khan, Naweed Gray, Alexander |
| contents | The task of policy compliance detection (PCD) is to determine if a scenario is in compliance with respect to a set of written policies. In a conversational setting, the results of PCD can indicate if clarifying questions must be asked to determine compliance status. Existing approaches usually claim to have reasoning capabilities that are latent or require a large amount of annotated data. In this work, we propose logical decomposition for policy compliance (LDPC): a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting. By selecting only a few exemplars alongside recently developed prompting techniques, we demonstrate that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies. The formulation of explicit logic graphs can in turn help answer PCDrelated questions with increased transparency and explainability. We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning. We also leverage the inherently interpretable architecture of LDPC to understand where errors occur, revealing ambiguities in the ShARC dataset and highlighting the challenges involved with reasoning for conversational question answering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_11335 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Few-shot Policy (de)composition in Conversational Question Answering Erwin, Kyle Axelrod, Guy Chang, Maria Fokoue, Achille Crouse, Maxwell Dan, Soham Gao, Tian Uceda-Sosa, Rosario Makondo, Ndivhuwo Khan, Naweed Gray, Alexander Computation and Language Artificial Intelligence The task of policy compliance detection (PCD) is to determine if a scenario is in compliance with respect to a set of written policies. In a conversational setting, the results of PCD can indicate if clarifying questions must be asked to determine compliance status. Existing approaches usually claim to have reasoning capabilities that are latent or require a large amount of annotated data. In this work, we propose logical decomposition for policy compliance (LDPC): a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting. By selecting only a few exemplars alongside recently developed prompting techniques, we demonstrate that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies. The formulation of explicit logic graphs can in turn help answer PCDrelated questions with increased transparency and explainability. We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning. We also leverage the inherently interpretable architecture of LDPC to understand where errors occur, revealing ambiguities in the ShARC dataset and highlighting the challenges involved with reasoning for conversational question answering. |
| title | Few-shot Policy (de)composition in Conversational Question Answering |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.11335 |