An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909574692339712 |
|---|---|
| author | Kaur, Navdeep McPheat, Lachlan Russo, Alessandra Cohn, Anthony G Madhyastha, Pranava |
| author_facet | Kaur, Navdeep McPheat, Lachlan Russo, Alessandra Cohn, Anthony G Madhyastha, Pranava |
| contents | In this paper, we examine the use of Conformal Language Modelling (CLM) alongside Answer Set Programming (ASP) to enhance the performance of standard open-weight LLMs on complex multi-step reasoning tasks. Using the StepGame dataset, which requires spatial reasoning, we apply CLM to generate sets of ASP programs from an LLM, providing statistical guarantees on the correctness of the outputs. Experimental results show that CLM significantly outperforms baseline models that use standard sampling methods, achieving substantial accuracy improvements across different levels of reasoning complexity. Additionally, the LLM-as-Judge metric enhances CLM's performance, especially in assessing structurally and logically correct ASP outputs. However, calibrating CLM with diverse calibration sets did not improve generalizability for tasks requiring much longer reasoning steps, indicating limitations in handling more complex tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05439 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning Kaur, Navdeep McPheat, Lachlan Russo, Alessandra Cohn, Anthony G Madhyastha, Pranava Computation and Language Artificial Intelligence In this paper, we examine the use of Conformal Language Modelling (CLM) alongside Answer Set Programming (ASP) to enhance the performance of standard open-weight LLMs on complex multi-step reasoning tasks. Using the StepGame dataset, which requires spatial reasoning, we apply CLM to generate sets of ASP programs from an LLM, providing statistical guarantees on the correctness of the outputs. Experimental results show that CLM significantly outperforms baseline models that use standard sampling methods, achieving substantial accuracy improvements across different levels of reasoning complexity. Additionally, the LLM-as-Judge metric enhances CLM's performance, especially in assessing structurally and logically correct ASP outputs. However, calibrating CLM with diverse calibration sets did not improve generalizability for tasks requiring much longer reasoning steps, indicating limitations in handling more complex tasks. |
| title | An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.05439 |