User Misconceptions of LLM-Based Conversational Programming Assistants
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917298098405376 |
|---|---|
| author | O'Brien, Gabrielle Alves, Antonio Pedro Santos Baltes, Sebastian Liebel, Grischa Lungu, Mircea Kalinowski, Marcos |
| author_facet | O'Brien, Gabrielle Alves, Antonio Pedro Santos Baltes, Sebastian Liebel, Grischa Lungu, Mircea Kalinowski, Marcos |
| contents | Programming assistants powered by large language models (LLMs) have become widely available, with conversational assistants like ChatGPT particularly accessible to novice programmers. However, varied tool capabilities and inconsistent availability of extensions (web search, code execution, retrieval-augmented generation) create opportunities for user misconceptions that may lead to over-reliance, unproductive practices, or insufficient quality control. We characterize misconceptions that users of conversational LLM-based assistants may have in programming contexts through a two-phase approach: first brainstorming and cataloging potential misconceptions, then conducting qualitative analysis of Python-programming conversations from the WildChat dataset. We find evidence that users have misplaced expectations about features like web access, code execution, and non-text outputs. We also note the potential for deeper conceptual issues around information requirements for debugging, validation, and optimization. Our findings reinforce the need for LLM-based tools to more clearly communicate their capabilities to users and empirically ground aspects that require clarification in programming contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25662 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | User Misconceptions of LLM-Based Conversational Programming Assistants O'Brien, Gabrielle Alves, Antonio Pedro Santos Baltes, Sebastian Liebel, Grischa Lungu, Mircea Kalinowski, Marcos Human-Computer Interaction Artificial Intelligence Programming assistants powered by large language models (LLMs) have become widely available, with conversational assistants like ChatGPT particularly accessible to novice programmers. However, varied tool capabilities and inconsistent availability of extensions (web search, code execution, retrieval-augmented generation) create opportunities for user misconceptions that may lead to over-reliance, unproductive practices, or insufficient quality control. We characterize misconceptions that users of conversational LLM-based assistants may have in programming contexts through a two-phase approach: first brainstorming and cataloging potential misconceptions, then conducting qualitative analysis of Python-programming conversations from the WildChat dataset. We find evidence that users have misplaced expectations about features like web access, code execution, and non-text outputs. We also note the potential for deeper conceptual issues around information requirements for debugging, validation, and optimization. Our findings reinforce the need for LLM-based tools to more clearly communicate their capabilities to users and empirically ground aspects that require clarification in programming contexts. |
| title | User Misconceptions of LLM-Based Conversational Programming Assistants |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2510.25662 |