ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913858883420160 |
|---|---|
| author | Yu, Botao Baker, Frazier N. Chen, Ziru Herb, Garrett Gou, Boyu Adu-Ampratwum, Daniel Ning, Xia Sun, Huan |
| author_facet | Yu, Botao Baker, Frazier N. Chen, Ziru Herb, Garrett Gou, Boyu Adu-Ampratwum, Daniel Ning, Xia Sun, Huan |
| contents | To enhance large language models (LLMs) for chemistry problem solving, several LLM-based agents augmented with tools have been proposed, such as ChemCrow and Coscientist. However, their evaluations are narrow in scope, leaving a large gap in understanding the benefits of tools across diverse chemistry tasks. To bridge this gap, we develop ChemToolAgent, an enhanced chemistry agent over ChemCrow, and conduct a comprehensive evaluation of its performance on both specialized chemistry tasks and general chemistry questions. Surprisingly, ChemToolAgent does not consistently outperform its base LLMs without tools. Our error analysis with a chemistry expert suggests that: For specialized chemistry tasks, such as synthesis prediction, we should augment agents with specialized tools; however, for general chemistry questions like those in exams, agents' ability to reason correctly with chemistry knowledge matters more, and tool augmentation does not always help. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07228 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving Yu, Botao Baker, Frazier N. Chen, Ziru Herb, Garrett Gou, Boyu Adu-Ampratwum, Daniel Ning, Xia Sun, Huan Artificial Intelligence Computational Engineering, Finance, and Science To enhance large language models (LLMs) for chemistry problem solving, several LLM-based agents augmented with tools have been proposed, such as ChemCrow and Coscientist. However, their evaluations are narrow in scope, leaving a large gap in understanding the benefits of tools across diverse chemistry tasks. To bridge this gap, we develop ChemToolAgent, an enhanced chemistry agent over ChemCrow, and conduct a comprehensive evaluation of its performance on both specialized chemistry tasks and general chemistry questions. Surprisingly, ChemToolAgent does not consistently outperform its base LLMs without tools. Our error analysis with a chemistry expert suggests that: For specialized chemistry tasks, such as synthesis prediction, we should augment agents with specialized tools; however, for general chemistry questions like those in exams, agents' ability to reason correctly with chemistry knowledge matters more, and tool augmentation does not always help. |
| title | ChemToolAgent: The Impact of Tools on Language Agents for Chemistry Problem Solving |
| topic | Artificial Intelligence Computational Engineering, Finance, and Science |
| url | https://arxiv.org/abs/2411.07228 |