FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913399856693248 |
|---|---|
| author | Xiao, Ruixuan Ma, Wentao Wang, Ke Wu, Yuchuan Zhao, Junbo Wang, Haobo Huang, Fei Li, Yongbin |
| author_facet | Xiao, Ruixuan Ma, Wentao Wang, Ke Wu, Yuchuan Zhao, Junbo Wang, Haobo Huang, Fei Li, Yongbin |
| contents | LLM-based agents have emerged as promising tools, which are crafted to fulfill complex tasks by iterative planning and action. However, these agents are susceptible to undesired planning hallucinations when lacking specific knowledge for expertise-intensive tasks. To address this, preliminary attempts are made to enhance planning reliability by incorporating external workflow-related knowledge. Despite the promise, such infused knowledge is mostly disorganized and diverse in formats, lacking rigorous formalization and comprehensive comparisons. Motivated by this, we formalize different formats of workflow knowledge and present FlowBench, the first benchmark for workflow-guided planning. FlowBench covers 51 different scenarios from 6 domains, with knowledge presented in diverse formats. To assess different LLMs on FlowBench, we design a multi-tiered evaluation framework. We evaluate the efficacy of workflow knowledge across multiple formats, and the results indicate that current LLM agents need considerable improvements for satisfactory planning. We hope that our challenging benchmark can pave the way for future agent planning research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14884 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents Xiao, Ruixuan Ma, Wentao Wang, Ke Wu, Yuchuan Zhao, Junbo Wang, Haobo Huang, Fei Li, Yongbin Computation and Language LLM-based agents have emerged as promising tools, which are crafted to fulfill complex tasks by iterative planning and action. However, these agents are susceptible to undesired planning hallucinations when lacking specific knowledge for expertise-intensive tasks. To address this, preliminary attempts are made to enhance planning reliability by incorporating external workflow-related knowledge. Despite the promise, such infused knowledge is mostly disorganized and diverse in formats, lacking rigorous formalization and comprehensive comparisons. Motivated by this, we formalize different formats of workflow knowledge and present FlowBench, the first benchmark for workflow-guided planning. FlowBench covers 51 different scenarios from 6 domains, with knowledge presented in diverse formats. To assess different LLMs on FlowBench, we design a multi-tiered evaluation framework. We evaluate the efficacy of workflow knowledge across multiple formats, and the results indicate that current LLM agents need considerable improvements for satisfactory planning. We hope that our challenging benchmark can pave the way for future agent planning research. |
| title | FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.14884 |