Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914895159623680 |
|---|---|
| author | Kazemitabaar, Majeed Williams, Jack Drosos, Ian Grossman, Tovi Henley, Austin Negreanu, Carina Sarkar, Advait |
| author_facet | Kazemitabaar, Majeed Williams, Jack Drosos, Ian Grossman, Tovi Henley, Austin Negreanu, Carina Sarkar, Advait |
| contents | LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle the challenging task of data analysis programming, which requires expertise in data processing, programming, and statistics. However, our formative study (n=15) uncovered serious challenges in verifying AI-generated results and steering the AI (i.e., guiding the AI system to produce the desired output). We developed two contrasting approaches to address these challenges. The first (Stepwise) decomposes the problem into step-by-step subgoals with pairs of editable assumptions and code until task completion, while the second (Phasewise) decomposes the entire problem into three editable, logical phases: structured input/output assumptions, execution plan, and code. A controlled, within-subjects experiment (n=18) compared these systems against a conversational baseline. Users reported significantly greater control with the Stepwise and Phasewise systems, and found intervention, correction, and verification easier, compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02651 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition Kazemitabaar, Majeed Williams, Jack Drosos, Ian Grossman, Tovi Henley, Austin Negreanu, Carina Sarkar, Advait Human-Computer Interaction Artificial Intelligence LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle the challenging task of data analysis programming, which requires expertise in data processing, programming, and statistics. However, our formative study (n=15) uncovered serious challenges in verifying AI-generated results and steering the AI (i.e., guiding the AI system to produce the desired output). We developed two contrasting approaches to address these challenges. The first (Stepwise) decomposes the problem into step-by-step subgoals with pairs of editable assumptions and code until task completion, while the second (Phasewise) decomposes the entire problem into three editable, logical phases: structured input/output assumptions, execution plan, and code. A controlled, within-subjects experiment (n=18) compared these systems against a conversational baseline. Users reported significantly greater control with the Stepwise and Phasewise systems, and found intervention, correction, and verification easier, compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools. |
| title | Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2407.02651 |