Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.05307 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910358534356992 |
|---|---|
| author | Li, Jinyang Huo, Nan Gao, Yan Shi, Jiayi Zhao, Yingxiu Qu, Ge Wu, Yurong Ma, Chenhao Lou, Jian-Guang Cheng, Reynold |
| author_facet | Li, Jinyang Huo, Nan Gao, Yan Shi, Jiayi Zhao, Yingxiu Qu, Ge Wu, Yurong Ma, Chenhao Lou, Jian-Guang Cheng, Reynold |
| contents | Interactive Data Analysis, the collaboration between humans and LLM agents, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task. To mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate LLM agents on interactive data analysis. Tapilot-Crossing contains 1024 interactions, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, Tapilot-Crossing is constructed by an economical multi-agent environment, Decision Company, with few human efforts. We evaluate popular and advanced LLM agents in Tapilot-Crossing, which underscores the challenges of interactive data analysis. Furthermore, we propose Adaptive Interaction Reflection (AIR), a self-generated reflection strategy that guides LLM agents to learn from successful history. Experiments demonstrate that Air can evolve LLMs into effective interactive data analysis agents, achieving a relative performance improvement of up to 44.5%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05307 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents Li, Jinyang Huo, Nan Gao, Yan Shi, Jiayi Zhao, Yingxiu Qu, Ge Wu, Yurong Ma, Chenhao Lou, Jian-Guang Cheng, Reynold Artificial Intelligence Interactive Data Analysis, the collaboration between humans and LLM agents, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task. To mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate LLM agents on interactive data analysis. Tapilot-Crossing contains 1024 interactions, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, Tapilot-Crossing is constructed by an economical multi-agent environment, Decision Company, with few human efforts. We evaluate popular and advanced LLM agents in Tapilot-Crossing, which underscores the challenges of interactive data analysis. Furthermore, we propose Adaptive Interaction Reflection (AIR), a self-generated reflection strategy that guides LLM agents to learn from successful history. Experiments demonstrate that Air can evolve LLMs into effective interactive data analysis agents, achieving a relative performance improvement of up to 44.5%. |
| title | Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2403.05307 |