Assessing and Verifying Task Utility in LLM-Powered Applications
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909199794962432 |
|---|---|
| author | Arabzadeh, Negar Huo, Siqing Mehta, Nikhil Wu, Qinqyun Wang, Chi Awadallah, Ahmed Clarke, Charles L. A. Kiseleva, Julia |
| author_facet | Arabzadeh, Negar Huo, Siqing Mehta, Nikhil Wu, Qinqyun Wang, Chi Awadallah, Ahmed Clarke, Charles L. A. Kiseleva, Julia |
| contents | The rapid development of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents, assisting humans in their daily tasks. However, a significant gap remains in assessing to what extent LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the need to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the effectiveness and robustness of AgentEval for two open source datasets including Math Problem solving and ALFWorld House-hold related tasks. For reproducibility purposes, we make the data, code and all the logs publicly available at https://bit.ly/3w3yKcS . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_02178 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Assessing and Verifying Task Utility in LLM-Powered Applications Arabzadeh, Negar Huo, Siqing Mehta, Nikhil Wu, Qinqyun Wang, Chi Awadallah, Ahmed Clarke, Charles L. A. Kiseleva, Julia Computation and Language Artificial Intelligence The rapid development of Large Language Models (LLMs) has led to a surge in applications that facilitate collaboration among multiple agents, assisting humans in their daily tasks. However, a significant gap remains in assessing to what extent LLM-powered applications genuinely enhance user experience and task execution efficiency. This highlights the need to verify utility of LLM-powered applications, particularly by ensuring alignment between the application's functionality and end-user needs. We introduce AgentEval, a novel framework designed to simplify the utility verification process by automatically proposing a set of criteria tailored to the unique purpose of any given application. This allows for a comprehensive assessment, quantifying the utility of an application against the suggested criteria. We present a comprehensive analysis of the effectiveness and robustness of AgentEval for two open source datasets including Math Problem solving and ALFWorld House-hold related tasks. For reproducibility purposes, we make the data, code and all the logs publicly available at https://bit.ly/3w3yKcS . |
| title | Assessing and Verifying Task Utility in LLM-Powered Applications |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.02178 |