Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2404.06411 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917635225026560 |
|---|---|
| author | Gioacchini, Luca Siracusano, Giuseppe Sanvito, Davide Gashteovski, Kiril Friede, David Bifulco, Roberto Lawrence, Carolin |
| author_facet | Gioacchini, Luca Siracusano, Giuseppe Sanvito, Davide Gashteovski, Kiril Friede, David Bifulco, Roberto Lawrence, Carolin |
| contents | The advances made by Large Language Models (LLMs) have led to the pursuit of LLM agents that can solve intricate, multi-step reasoning tasks. As with any research pursuit, benchmarking and evaluation are key corner stones to efficient and reliable progress. However, existing benchmarks are often narrow and simply compute overall task success. To face these issues, we propose AgentQuest -- a framework where (i) both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs; (ii) we offer two new evaluation metrics that can reliably track LLM agent progress while solving a task. We exemplify the utility of the metrics on two use cases wherein we identify common failure points and refine the agent architecture to obtain a significant performance increase. Together with the research community, we hope to extend AgentQuest further and therefore we make it available under https://github.com/nec-research/agentquest. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06411 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents Gioacchini, Luca Siracusano, Giuseppe Sanvito, Davide Gashteovski, Kiril Friede, David Bifulco, Roberto Lawrence, Carolin Artificial Intelligence Computation and Language The advances made by Large Language Models (LLMs) have led to the pursuit of LLM agents that can solve intricate, multi-step reasoning tasks. As with any research pursuit, benchmarking and evaluation are key corner stones to efficient and reliable progress. However, existing benchmarks are often narrow and simply compute overall task success. To face these issues, we propose AgentQuest -- a framework where (i) both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs; (ii) we offer two new evaluation metrics that can reliably track LLM agent progress while solving a task. We exemplify the utility of the metrics on two use cases wherein we identify common failure points and refine the agent architecture to obtain a significant performance increase. Together with the research community, we hope to extend AgentQuest further and therefore we make it available under https://github.com/nec-research/agentquest. |
| title | AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2404.06411 |