Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911990743564288 |
|---|---|
| author | Kumar, Harsh Reza, Mohi Mitchell, Jeb Musabirov, Ilya Zhang, Lisa Liut, Michael |
| author_facet | Kumar, Harsh Reza, Mohi Mitchell, Jeb Musabirov, Ilya Zhang, Lisa Liut, Michael |
| contents | Growth in the use of large language models (LLMs) in programming education is altering how students write SQL queries. Traditionally, students relied heavily on web search for coding assistance, but this has shifted with the adoption of LLMs like ChatGPT. However, the comparative process and outcomes of using web search versus LLMs for coding help remain underexplored. To address this, we conducted a randomized interview study in a database classroom to compare web search and LLMs, including a publicly available LLM (ChatGPT) and an instructor-tuned LLM, for writing SQL queries. Our findings indicate that using an instructor-tuned LLM required significantly more interactions than both ChatGPT and web search, but resulted in a similar number of edits to the final SQL query. No significant differences were found in the quality of the final SQL queries between conditions, although the LLM conditions directionally showed higher query quality. Furthermore, students using instructor-tuned LLM reported a lower mental demand. These results have implications for learning and productivity in programming education. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08401 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries Kumar, Harsh Reza, Mohi Mitchell, Jeb Musabirov, Ilya Zhang, Lisa Liut, Michael Human-Computer Interaction Artificial Intelligence Computers and Society Databases Growth in the use of large language models (LLMs) in programming education is altering how students write SQL queries. Traditionally, students relied heavily on web search for coding assistance, but this has shifted with the adoption of LLMs like ChatGPT. However, the comparative process and outcomes of using web search versus LLMs for coding help remain underexplored. To address this, we conducted a randomized interview study in a database classroom to compare web search and LLMs, including a publicly available LLM (ChatGPT) and an instructor-tuned LLM, for writing SQL queries. Our findings indicate that using an instructor-tuned LLM required significantly more interactions than both ChatGPT and web search, but resulted in a similar number of edits to the final SQL query. No significant differences were found in the quality of the final SQL queries between conditions, although the LLM conditions directionally showed higher query quality. Furthermore, students using instructor-tuned LLM reported a lower mental demand. These results have implications for learning and productivity in programming education. |
| title | Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries |
| topic | Human-Computer Interaction Artificial Intelligence Computers and Society Databases |
| url | https://arxiv.org/abs/2408.08401 |