Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917679629074432 |
|---|---|
| author | Lal, Yash Kumar Zhang, Li Brahman, Faeze Majumder, Bodhisattwa Prasad Clark, Peter Tandon, Niket |
| author_facet | Lal, Yash Kumar Zhang, Li Brahman, Faeze Majumder, Bodhisattwa Prasad Clark, Peter Tandon, Niket |
| contents | How-to procedures, such as how to plant a garden, are now used by millions of users, but sometimes need customizing to meet a user's specific needs, e.g., planting a garden without pesticides. Our goal is to measure and improve an LLM's ability to perform such customization. Our approach is to test several simple multi-LLM-agent architectures for customization, as well as an end-to-end LLM, using a new evaluation set, called CustomPlans, of over 200 WikiHow procedures each with a customization need. We find that a simple architecture with two LLM agents used sequentially performs best, one that edits a generic how-to procedure and one that verifies its executability, significantly outperforming (10.5% absolute) an end-to-end prompted LLM. This suggests that LLMs can be configured reasonably effectively for procedure customization. This also suggests that multi-agent editing architectures may be worth exploring further for other customization applications (e.g. coding, creative writing) in the future. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09510 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization Lal, Yash Kumar Zhang, Li Brahman, Faeze Majumder, Bodhisattwa Prasad Clark, Peter Tandon, Niket Computation and Language How-to procedures, such as how to plant a garden, are now used by millions of users, but sometimes need customizing to meet a user's specific needs, e.g., planting a garden without pesticides. Our goal is to measure and improve an LLM's ability to perform such customization. Our approach is to test several simple multi-LLM-agent architectures for customization, as well as an end-to-end LLM, using a new evaluation set, called CustomPlans, of over 200 WikiHow procedures each with a customization need. We find that a simple architecture with two LLM agents used sequentially performs best, one that edits a generic how-to procedure and one that verifies its executability, significantly outperforming (10.5% absolute) an end-to-end prompted LLM. This suggests that LLMs can be configured reasonably effectively for procedure customization. This also suggests that multi-agent editing architectures may be worth exploring further for other customization applications (e.g. coding, creative writing) in the future. |
| title | Tailoring with Targeted Precision: Edit-Based Agents for Open-Domain Procedure Customization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.09510 |