How Beginning Programmers and Code LLMs (Mis)read Each Other
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914860299714560 |
|---|---|
| author | Nguyen, Sydney Babe, Hannah McLean Zi, Yangtian Guha, Arjun Anderson, Carolyn Jane Feldman, Molly Q |
| author_facet | Nguyen, Sydney Babe, Hannah McLean Zi, Yangtian Guha, Arjun Anderson, Carolyn Jane Feldman, Molly Q |
| contents | Generative AI models, specifically large language models (LLMs), have made strides towards the long-standing goal of text-to-code generation. This progress has invited numerous studies of user interaction. However, less is known about the struggles and strategies of non-experts, for whom each step of the text-to-code problem presents challenges: describing their intent in natural language, evaluating the correctness of generated code, and editing prompts when the generated code is incorrect. This paper presents a large-scale controlled study of how 120 beginning coders across three academic institutions approach writing and editing prompts. A novel experimental design allows us to target specific steps in the text-to-code process and reveals that beginners struggle with writing and editing prompts, even for problems at their skill level and when correctness is automatically determined. Our mixed-methods evaluation provides insight into student processes and perceptions with key implications for non-expert Code LLM use within and outside of education. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_15232 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How Beginning Programmers and Code LLMs (Mis)read Each Other Nguyen, Sydney Babe, Hannah McLean Zi, Yangtian Guha, Arjun Anderson, Carolyn Jane Feldman, Molly Q Human-Computer Interaction Generative AI models, specifically large language models (LLMs), have made strides towards the long-standing goal of text-to-code generation. This progress has invited numerous studies of user interaction. However, less is known about the struggles and strategies of non-experts, for whom each step of the text-to-code problem presents challenges: describing their intent in natural language, evaluating the correctness of generated code, and editing prompts when the generated code is incorrect. This paper presents a large-scale controlled study of how 120 beginning coders across three academic institutions approach writing and editing prompts. A novel experimental design allows us to target specific steps in the text-to-code process and reveals that beginners struggle with writing and editing prompts, even for problems at their skill level and when correctness is automatically determined. Our mixed-methods evaluation provides insight into student processes and perceptions with key implications for non-expert Code LLM use within and outside of education. |
| title | How Beginning Programmers and Code LLMs (Mis)read Each Other |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2401.15232 |