TOOLVERIFIER: Generalization to New Tools via Self-Verification
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917612596756480 |
|---|---|
| author | Mekala, Dheeraj Weston, Jason Lanchantin, Jack Raileanu, Roberta Lomeli, Maria Shang, Jingbo Dwivedi-Yu, Jane |
| author_facet | Mekala, Dheeraj Weston, Jason Lanchantin, Jack Raileanu, Roberta Lomeli, Maria Shang, Jingbo Dwivedi-Yu, Jane |
| contents | Teaching language models to use tools is an important milestone towards building general assistants, but remains an open problem. While there has been significant progress on learning to use specific tools via fine-tuning, language models still struggle with learning how to robustly use new tools from only a few demonstrations. In this work we introduce a self-verification method which distinguishes between close candidates by self-asking contrastive questions during (1) tool selection; and (2) parameter generation. We construct synthetic, high-quality, self-generated data for this goal using Llama-2 70B, which we intend to release publicly. Extensive experiments on 4 tasks from the ToolBench benchmark, consisting of 17 unseen tools, demonstrate an average improvement of 22% over few-shot baselines, even in scenarios where the distinctions between candidate tools are finely nuanced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14158 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TOOLVERIFIER: Generalization to New Tools via Self-Verification Mekala, Dheeraj Weston, Jason Lanchantin, Jack Raileanu, Roberta Lomeli, Maria Shang, Jingbo Dwivedi-Yu, Jane Computation and Language Teaching language models to use tools is an important milestone towards building general assistants, but remains an open problem. While there has been significant progress on learning to use specific tools via fine-tuning, language models still struggle with learning how to robustly use new tools from only a few demonstrations. In this work we introduce a self-verification method which distinguishes between close candidates by self-asking contrastive questions during (1) tool selection; and (2) parameter generation. We construct synthetic, high-quality, self-generated data for this goal using Llama-2 70B, which we intend to release publicly. Extensive experiments on 4 tasks from the ToolBench benchmark, consisting of 17 unseen tools, demonstrate an average improvement of 22% over few-shot baselines, even in scenarios where the distinctions between candidate tools are finely nuanced. |
| title | TOOLVERIFIER: Generalization to New Tools via Self-Verification |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.14158 |