DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909155530375168 |
|---|---|
| author | Mishra, Manit Braham, Abderrahman Marsom, Charles Chung, Bryan Griffin, Gavin Sidnerlikar, Dakshesh Sarin, Chatanya Rajaram, Arjun |
| author_facet | Mishra, Manit Braham, Abderrahman Marsom, Charles Chung, Bryan Griffin, Gavin Sidnerlikar, Dakshesh Sarin, Chatanya Rajaram, Arjun |
| contents | Conventional processes for analyzing datasets and extracting meaningful information are often time-consuming and laborious. Previous work has identified manual, repetitive coding and data collection as major obstacles that hinder data scientists from undertaking more nuanced labor and high-level projects. To combat this, we evaluated OpenAI's GPT-3.5 as a "Language Data Scientist" (LDS) that can extrapolate key findings, including correlations and basic information, from a given dataset. The model was tested on a diverse set of benchmark datasets to evaluate its performance across multiple standards, including data science code-generation based tasks involving libraries such as NumPy, Pandas, Scikit-Learn, and TensorFlow, and was broadly successful in correctly answering a given data science query related to the benchmark dataset. The LDS used various novel prompt engineering techniques to effectively answer a given question, including Chain-of-Thought reinforcement and SayCan prompt engineering. Our findings demonstrate great potential for leveraging Large Language Models for low-level, zero-shot data analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_00188 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries Mishra, Manit Braham, Abderrahman Marsom, Charles Chung, Bryan Griffin, Gavin Sidnerlikar, Dakshesh Sarin, Chatanya Rajaram, Arjun Computation and Language Artificial Intelligence Conventional processes for analyzing datasets and extracting meaningful information are often time-consuming and laborious. Previous work has identified manual, repetitive coding and data collection as major obstacles that hinder data scientists from undertaking more nuanced labor and high-level projects. To combat this, we evaluated OpenAI's GPT-3.5 as a "Language Data Scientist" (LDS) that can extrapolate key findings, including correlations and basic information, from a given dataset. The model was tested on a diverse set of benchmark datasets to evaluate its performance across multiple standards, including data science code-generation based tasks involving libraries such as NumPy, Pandas, Scikit-Learn, and TensorFlow, and was broadly successful in correctly answering a given data science query related to the benchmark dataset. The LDS used various novel prompt engineering techniques to effectively answer a given question, including Chain-of-Thought reinforcement and SayCan prompt engineering. Our findings demonstrate great potential for leveraging Large Language Models for low-level, zero-shot data analysis. |
| title | DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.00188 |