AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909944849104896 |
|---|---|
| author | Kim, Takyoung Singh, Janvijay Mehri, Shuhaib Acikgoz, Emre Can Mukherjee, Sagnik Bozdag, Nimet Beyza Shashidhar, Sumuk Tur, Gokhan Hakkani-Tür, Dilek |
| author_facet | Kim, Takyoung Singh, Janvijay Mehri, Shuhaib Acikgoz, Emre Can Mukherjee, Sagnik Bozdag, Nimet Beyza Shashidhar, Sumuk Tur, Gokhan Hakkani-Tür, Dilek |
| contents | The growing capabilities of large language models (LLMs) in instruction-following and context-understanding lead to the era of agents with numerous applications. Among these, task planning agents have become especially prominent in realistic scenarios involving complex internal pipelines, such as context understanding, tool management, and response generation. However, existing benchmarks predominantly evaluate agent performance based on task completion as a proxy for overall effectiveness. We hypothesize that merely improving task completion is misaligned with maximizing user satisfaction, as users interact with the entire agentic process and not only the end result. To address this gap, we propose AURA, an Agent-User inteRaction Assessment framework that conceptualizes the behavioral stages of interactive task planning agents. AURA offers a comprehensive assessment of agent through a set of atomic LLM evaluation criteria, allowing researchers and practitioners to diagnose specific strengths and weaknesses within the agent's decision-making pipeline. Our analyses show that agents excel in different behavioral stages, with user satisfaction shaped by both outcomes and intermediate behaviors. We also highlight future directions, including systems that leverage multiple agents and the limitations of user simulators in task planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_01592 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents Kim, Takyoung Singh, Janvijay Mehri, Shuhaib Acikgoz, Emre Can Mukherjee, Sagnik Bozdag, Nimet Beyza Shashidhar, Sumuk Tur, Gokhan Hakkani-Tür, Dilek Computation and Language Artificial Intelligence The growing capabilities of large language models (LLMs) in instruction-following and context-understanding lead to the era of agents with numerous applications. Among these, task planning agents have become especially prominent in realistic scenarios involving complex internal pipelines, such as context understanding, tool management, and response generation. However, existing benchmarks predominantly evaluate agent performance based on task completion as a proxy for overall effectiveness. We hypothesize that merely improving task completion is misaligned with maximizing user satisfaction, as users interact with the entire agentic process and not only the end result. To address this gap, we propose AURA, an Agent-User inteRaction Assessment framework that conceptualizes the behavioral stages of interactive task planning agents. AURA offers a comprehensive assessment of agent through a set of atomic LLM evaluation criteria, allowing researchers and practitioners to diagnose specific strengths and weaknesses within the agent's decision-making pipeline. Our analyses show that agents excel in different behavioral stages, with user satisfaction shaped by both outcomes and intermediate behaviors. We also highlight future directions, including systems that leverage multiple agents and the limitations of user simulators in task planning. |
| title | AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.01592 |