Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911782624296960 |
|---|---|
| author | Arabzadeh, Negar Kiseleva, Julia Wu, Qingyun Wang, Chi Awadallah, Ahmed Dibia, Victor Fourney, Adam Clarke, Charles |
| author_facet | Arabzadeh, Negar Kiseleva, Julia Wu, Qingyun Wang, Chi Awadallah, Ahmed Dibia, Victor Fourney, Adam Clarke, Charles |
| contents | The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks. However, a significant gap remains in assessing whether LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the pressing need for methods to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval provides an implementation for the math problems, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the robustness of quantifier's work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_09015 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications Arabzadeh, Negar Kiseleva, Julia Wu, Qingyun Wang, Chi Awadallah, Ahmed Dibia, Victor Fourney, Adam Clarke, Charles Computation and Language Artificial Intelligence The rapid development in the field of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents to assist humans in their daily tasks. However, a significant gap remains in assessing whether LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the pressing need for methods to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval provides an implementation for the math problems, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the robustness of quantifier's work. |
| title | Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applications |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.09015 |