From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916707864412160 |
|---|---|
| author | Alfageeh, Ali Zarkouei, Sadegh AlMahdi Kazemi Nam, Daye Prol, Daniel Amoozadeh, Matin Chattopadhyay, Souti Prather, James Denny, Paul Leinonen, Juho Hilton, Michael Ragavan, Sruti Srinivasa Alipour, Mohammad Amin |
| author_facet | Alfageeh, Ali Zarkouei, Sadegh AlMahdi Kazemi Nam, Daye Prol, Daniel Amoozadeh, Matin Chattopadhyay, Souti Prather, James Denny, Paul Leinonen, Juho Hilton, Michael Ragavan, Sruti Srinivasa Alipour, Mohammad Amin |
| contents | Background and Context. The increasing integration of large language models (LLMs) in computing education presents an emerging challenge in understanding how students use LLMs and craft prompts to solve computational tasks. Prior research has used both qualitative and quantitative methods to analyze prompting behavior, but these approaches lack scalability or fail to effectively capture the semantic evolution of prompts. Objective. In this paper, we investigate whether students prompts can be systematically analyzed using propositional logic constraints. We examine whether this approach can identify patterns in prompt evolution, detect struggling students, and provide insights into effective and ineffective strategies. Method. We introduce Prompt2Constraints, a novel method that translates students prompts into logical constraints. The constraints are able to represent the intent of the prompts in succinct and quantifiable ways. We used this approach to analyze a dataset of 1,872 prompts from 203 students solving introductory programming tasks. Findings. We find that while successful and unsuccessful attempts tend to use a similar number of constraints overall, when students fail, they often modify their prompts more significantly, shifting problem-solving strategies midway. We also identify points where specific interventions could be most helpful to students for refining their prompts. Implications. This work offers a new and scalable way to detect students who struggle in solving natural language programming tasks. This work could be extended to investigate more complex tasks and integrated into programming tools to provide real-time support. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_18691 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions Alfageeh, Ali Zarkouei, Sadegh AlMahdi Kazemi Nam, Daye Prol, Daniel Amoozadeh, Matin Chattopadhyay, Souti Prather, James Denny, Paul Leinonen, Juho Hilton, Michael Ragavan, Sruti Srinivasa Alipour, Mohammad Amin Human-Computer Interaction Artificial Intelligence Software Engineering Background and Context. The increasing integration of large language models (LLMs) in computing education presents an emerging challenge in understanding how students use LLMs and craft prompts to solve computational tasks. Prior research has used both qualitative and quantitative methods to analyze prompting behavior, but these approaches lack scalability or fail to effectively capture the semantic evolution of prompts. Objective. In this paper, we investigate whether students prompts can be systematically analyzed using propositional logic constraints. We examine whether this approach can identify patterns in prompt evolution, detect struggling students, and provide insights into effective and ineffective strategies. Method. We introduce Prompt2Constraints, a novel method that translates students prompts into logical constraints. The constraints are able to represent the intent of the prompts in succinct and quantifiable ways. We used this approach to analyze a dataset of 1,872 prompts from 203 students solving introductory programming tasks. Findings. We find that while successful and unsuccessful attempts tend to use a similar number of constraints overall, when students fail, they often modify their prompts more significantly, shifting problem-solving strategies midway. We also identify points where specific interventions could be most helpful to students for refining their prompts. Implications. This work offers a new and scalable way to detect students who struggle in solving natural language programming tasks. This work could be extended to investigate more complex tasks and integrated into programming tools to provide real-time support. |
| title | From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions |
| topic | Human-Computer Interaction Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2504.18691 |