Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.11886 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909222844760064 |
|---|---|
| author | Huang, Xiang Cheng, Sitao Huang, Shanshan Shen, Jiayu Xu, Yong Zhang, Chaoyun Qu, Yuzhong |
| author_facet | Huang, Xiang Cheng, Sitao Huang, Shanshan Shen, Jiayu Xu, Yong Zhang, Chaoyun Qu, Yuzhong |
| contents | Employing Large Language Models (LLMs) for semantic parsing has achieved remarkable success. However, we find existing methods fall short in terms of reliability and efficiency when hallucinations are encountered. In this paper, we address these challenges with a framework called QueryAgent, which solves a question step-by-step and performs step-wise self-correction. We introduce an environmental feedback-based self-correction method called ERASER. Unlike traditional approaches, ERASER leverages rich environmental feedback in the intermediate steps to perform selective and differentiated self-correction only when necessary. Experimental results demonstrate that QueryAgent notably outperforms all previous few-shot methods using only one example on GrailQA and GraphQ by 7.0 and 15.0 F1. Moreover, our approach exhibits superiority in terms of efficiency, including runtime, query overhead, and API invocation costs. By leveraging ERASER, we further improve another baseline (i.e., AgentBench) by approximately 10 points, revealing the strong transferability of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_11886 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback-based Self-Correction Huang, Xiang Cheng, Sitao Huang, Shanshan Shen, Jiayu Xu, Yong Zhang, Chaoyun Qu, Yuzhong Computation and Language Artificial Intelligence Employing Large Language Models (LLMs) for semantic parsing has achieved remarkable success. However, we find existing methods fall short in terms of reliability and efficiency when hallucinations are encountered. In this paper, we address these challenges with a framework called QueryAgent, which solves a question step-by-step and performs step-wise self-correction. We introduce an environmental feedback-based self-correction method called ERASER. Unlike traditional approaches, ERASER leverages rich environmental feedback in the intermediate steps to perform selective and differentiated self-correction only when necessary. Experimental results demonstrate that QueryAgent notably outperforms all previous few-shot methods using only one example on GrailQA and GraphQ by 7.0 and 15.0 F1. Moreover, our approach exhibits superiority in terms of efficiency, including runtime, query overhead, and API invocation costs. By leveraging ERASER, we further improve another baseline (i.e., AgentBench) by approximately 10 points, revealing the strong transferability of our approach. |
| title | QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback-based Self-Correction |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.11886 |