Substance Beats Style: Why Beginning Students Fail to Code with LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929561085673472 |
|---|---|
| author | Lucchetti, Francesca Wu, Zixuan Guha, Arjun Feldman, Molly Q Anderson, Carolyn Jane |
| author_facet | Lucchetti, Francesca Wu, Zixuan Guha, Arjun Feldman, Molly Q Anderson, Carolyn Jane |
| contents | Although LLMs are increasing the productivity of professional programmers, existing work shows that beginners struggle to prompt LLMs to solve text-to-code tasks. Why is this the case? This paper explores two competing hypotheses about the cause of student-LLM miscommunication: (1) students simply lack the technical vocabulary needed to write good prompts, and (2) students do not understand the extent of information that LLMs need to solve code generation tasks. We study (1) with a causal intervention experiment on technical vocabulary and (2) by analyzing graphs that abstract how students edit prompts and the different failures that they encounter. We find that substance beats style: a poor grasp of technical vocabulary is merely correlated with prompt failure; that the information content of prompts predicts success; that students get stuck making trivial edits; and more. Our findings have implications for the use of LLMs in programming education, and for efforts to make computing more accessible with LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19792 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Substance Beats Style: Why Beginning Students Fail to Code with LLMs Lucchetti, Francesca Wu, Zixuan Guha, Arjun Feldman, Molly Q Anderson, Carolyn Jane Computers and Society Machine Learning Although LLMs are increasing the productivity of professional programmers, existing work shows that beginners struggle to prompt LLMs to solve text-to-code tasks. Why is this the case? This paper explores two competing hypotheses about the cause of student-LLM miscommunication: (1) students simply lack the technical vocabulary needed to write good prompts, and (2) students do not understand the extent of information that LLMs need to solve code generation tasks. We study (1) with a causal intervention experiment on technical vocabulary and (2) by analyzing graphs that abstract how students edit prompts and the different failures that they encounter. We find that substance beats style: a poor grasp of technical vocabulary is merely correlated with prompt failure; that the information content of prompts predicts success; that students get stuck making trivial edits; and more. Our findings have implications for the use of LLMs in programming education, and for efforts to make computing more accessible with LLMs. |
| title | Substance Beats Style: Why Beginning Students Fail to Code with LLMs |
| topic | Computers and Society Machine Learning |
| url | https://arxiv.org/abs/2410.19792 |