Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929464579981312 |
|---|---|
| author | Kumar, Harsh Musabirov, Ilya Reza, Mohi Shi, Jiakai Wang, Xinyuan Williams, Joseph Jay Kuzminykh, Anastasia Liut, Michael |
| author_facet | Kumar, Harsh Musabirov, Ilya Reza, Mohi Shi, Jiakai Wang, Xinyuan Williams, Joseph Jay Kuzminykh, Anastasia Liut, Michael |
| contents | Personalized chatbot-based teaching assistants can be crucial in addressing increasing classroom sizes, especially where direct teacher presence is limited. Large language models (LLMs) offer a promising avenue, with increasing research exploring their educational utility. However, the challenge lies not only in establishing the efficacy of LLMs but also in discerning the nuances of interaction between learners and these models, which impact learners' engagement and results. We conducted a formative study in an undergraduate computer science classroom (N=145) and a controlled experiment on Prolific (N=356) to explore the impact of four pedagogically informed guidance strategies on the learners' performance, confidence and trust in LLMs. Direct LLM answers marginally improved performance, while refining student solutions fostered trust. Structured guidance reduced random queries as well as instances of students copy-pasting assignment questions to the LLM. Our work highlights the role that teachers can play in shaping LLM-supported learning environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_13712 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception Kumar, Harsh Musabirov, Ilya Reza, Mohi Shi, Jiakai Wang, Xinyuan Williams, Joseph Jay Kuzminykh, Anastasia Liut, Michael Human-Computer Interaction Artificial Intelligence Personalized chatbot-based teaching assistants can be crucial in addressing increasing classroom sizes, especially where direct teacher presence is limited. Large language models (LLMs) offer a promising avenue, with increasing research exploring their educational utility. However, the challenge lies not only in establishing the efficacy of LLMs but also in discerning the nuances of interaction between learners and these models, which impact learners' engagement and results. We conducted a formative study in an undergraduate computer science classroom (N=145) and a controlled experiment on Prolific (N=356) to explore the impact of four pedagogically informed guidance strategies on the learners' performance, confidence and trust in LLMs. Direct LLM answers marginally improved performance, while refining student solutions fostered trust. Structured guidance reduced random queries as well as instances of students copy-pasting assignment questions to the LLM. Our work highlights the role that teachers can play in shaping LLM-supported learning environments. |
| title | Impact of Guidance and Interaction Strategies for LLM Use on Learner Performance and Perception |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2310.13712 |