Tools Fail: Detecting Silent Errors in Faulty Tools
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910504542273536 |
|---|---|
| author | Sun, Jimin Min, So Yeon Chang, Yingshan Bisk, Yonatan |
| author_facet | Sun, Jimin Min, So Yeon Chang, Yingshan Bisk, Yonatan |
| contents | Tools have become a mainstay of LLMs, allowing them to retrieve knowledge not in their weights, to perform tasks on the web, and even to control robots. However, most ontologies and surveys of tool-use have assumed the core challenge for LLMs is choosing the tool. Instead, we introduce a framework for tools more broadly which guides us to explore a model's ability to detect "silent" tool errors, and reflect on how to plan. This more directly aligns with the increasingly popular use of models as tools. We provide an initial approach to failure recovery with promising results both on a controlled calculator setting and embodied agent planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19228 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tools Fail: Detecting Silent Errors in Faulty Tools Sun, Jimin Min, So Yeon Chang, Yingshan Bisk, Yonatan Computation and Language Artificial Intelligence Machine Learning Tools have become a mainstay of LLMs, allowing them to retrieve knowledge not in their weights, to perform tasks on the web, and even to control robots. However, most ontologies and surveys of tool-use have assumed the core challenge for LLMs is choosing the tool. Instead, we introduce a framework for tools more broadly which guides us to explore a model's ability to detect "silent" tool errors, and reflect on how to plan. This more directly aligns with the increasingly popular use of models as tools. We provide an initial approach to failure recovery with promising results both on a controlled calculator setting and embodied agent planning. |
| title | Tools Fail: Detecting Silent Errors in Faulty Tools |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.19228 |