Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.16551 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917502483693568 |
|---|---|
| author | Lu, Yunan Liu, Luigi Yahia, Omar Sharma, Arpit Yu, Zhou |
| author_facet | Lu, Yunan Liu, Luigi Yahia, Omar Sharma, Arpit Yu, Zhou |
| contents | Evaluating LLM-based agents remains challenging because identifying meaningful failure cases often requires substantial human effort to design realistic test scenarios. Prior works primarily focus on automatically discovering agent failures induced by adversarial users, while overlooking queries with real user intents that also trigger agent failures. We introduce PQR, a framework that not only surfaces agent failures with respect to specific objectives (e.g., helpfulness, safety, etc.) but also resembles real users' intents. PQR operates through an iterative interaction between two complementary modules. The query refinement module performs rewrites to explore diverse query variations, while the prompt refinement module uses prior feedback to derive new objective-violating strategies and realism policies for refining prompts, which in turn generate failure-triggering yet realistic queries. We evaluate PQR on detecting an e-commerce QA agent's unhelpful responses. Our method uncovers 23% - 78% more unhelpful responses, and our generated queries are more diverse and realistic compared to previous methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_16551 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures Lu, Yunan Liu, Luigi Yahia, Omar Sharma, Arpit Yu, Zhou Computation and Language Evaluating LLM-based agents remains challenging because identifying meaningful failure cases often requires substantial human effort to design realistic test scenarios. Prior works primarily focus on automatically discovering agent failures induced by adversarial users, while overlooking queries with real user intents that also trigger agent failures. We introduce PQR, a framework that not only surfaces agent failures with respect to specific objectives (e.g., helpfulness, safety, etc.) but also resembles real users' intents. PQR operates through an iterative interaction between two complementary modules. The query refinement module performs rewrites to explore diverse query variations, while the prompt refinement module uses prior feedback to derive new objective-violating strategies and realism policies for refining prompts, which in turn generate failure-triggering yet realistic queries. We evaluate PQR on detecting an e-commerce QA agent's unhelpful responses. Our method uncovers 23% - 78% more unhelpful responses, and our generated queries are more diverse and realistic compared to previous methods. |
| title | PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.16551 |