$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914837730164736 |
|---|---|
| author | Yao, Shunyu Shinn, Noah Razavi, Pedram Narasimhan, Karthik |
| author_facet | Yao, Shunyu Shinn, Noah Razavi, Pedram Narasimhan, Karthik |
| contents | Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $τ$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12045 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains Yao, Shunyu Shinn, Noah Razavi, Pedram Narasimhan, Karthik Artificial Intelligence Computation and Language Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world applications. We propose $τ$-bench, a benchmark emulating dynamic conversations between a user (simulated by language models) and a language agent provided with domain-specific API tools and policy guidelines. We employ an efficient and faithful evaluation process that compares the database state at the end of a conversation with the annotated goal state. We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials. Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail). Our findings point to the need for methods that can improve the ability of agents to act consistently and follow rules reliably. |
| title | $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.12045 |