CIBench: Evaluating Your LLMs with a Code Interpreter Plugin
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909378783739904 |
|---|---|
| author | Zhang, Chuyu Zhang, Songyang Hu, Yingfan Shen, Haowen Liu, Kuikun Ma, Zerun Zhou, Fengzhe Zhang, Wenwei He, Xuming Lin, Dahua Chen, Kai |
| author_facet | Zhang, Chuyu Zhang, Songyang Hu, Yingfan Shen, Haowen Liu, Kuikun Ma, Zerun Zhou, Fengzhe Zhang, Wenwei He, Xuming Lin, Dahua Chen, Kai |
| contents | While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this paper, we propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. Our evaluation framework includes an evaluation dataset and two evaluation modes. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. The two evaluation modes assess LLMs' ability with and without human assistance. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_10499 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CIBench: Evaluating Your LLMs with a Code Interpreter Plugin Zhang, Chuyu Zhang, Songyang Hu, Yingfan Shen, Haowen Liu, Kuikun Ma, Zerun Zhou, Fengzhe Zhang, Wenwei He, Xuming Lin, Dahua Chen, Kai Computation and Language While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this paper, we propose an interactive evaluation framework, named CIBench, to comprehensively assess LLMs' ability to utilize code interpreters for data science tasks. Our evaluation framework includes an evaluation dataset and two evaluation modes. The evaluation dataset is constructed using an LLM-human cooperative approach and simulates an authentic workflow by leveraging consecutive and interactive IPython sessions. The two evaluation modes assess LLMs' ability with and without human assistance. We conduct extensive experiments to analyze the ability of 24 LLMs on CIBench and provide valuable insights for future LLMs in code interpreter utilization. |
| title | CIBench: Evaluating Your LLMs with a Code Interpreter Plugin |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.10499 |