LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929593271713792 |
|---|---|
| author | Leinonen, Juho Denny, Paul Kiljunen, Olli MacNeil, Stephen Sarsa, Sami Hellas, Arto |
| author_facet | Leinonen, Juho Denny, Paul Kiljunen, Olli MacNeil, Stephen Sarsa, Sami Hellas, Arto |
| contents | There is a great need for data in computing education research. Data is needed to understand how students behave, to train models of student behavior to optimally support students, and to develop and validate new assessment tools and learning analytics techniques. However, relatively few computing education datasets are shared openly, often due to privacy regulations and issues in making sure the data is anonymous. Large language models (LLMs) offer a promising approach to create large-scale, privacy-preserving synthetic data, which can be used to explore various aspects of student learning, develop and test educational technologies, and support research in areas where collecting real student data may be challenging or impractical. This work explores generating synthetic buggy code submissions for introductory programming exercises using GPT-4o. We compare the distribution of test case failures between synthetic and real student data from two courses to analyze the accuracy of the synthetic data in mimicking real student data. Our findings suggest that LLMs can be used to generate synthetic incorrect submissions that are not significantly different from real student data with regard to test case failure distributions. Our research contributes to the development of reliable synthetic datasets for computing education research and teaching, potentially accelerating progress in the field while preserving student privacy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10455 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education Leinonen, Juho Denny, Paul Kiljunen, Olli MacNeil, Stephen Sarsa, Sami Hellas, Arto Computers and Society Software Engineering There is a great need for data in computing education research. Data is needed to understand how students behave, to train models of student behavior to optimally support students, and to develop and validate new assessment tools and learning analytics techniques. However, relatively few computing education datasets are shared openly, often due to privacy regulations and issues in making sure the data is anonymous. Large language models (LLMs) offer a promising approach to create large-scale, privacy-preserving synthetic data, which can be used to explore various aspects of student learning, develop and test educational technologies, and support research in areas where collecting real student data may be challenging or impractical. This work explores generating synthetic buggy code submissions for introductory programming exercises using GPT-4o. We compare the distribution of test case failures between synthetic and real student data from two courses to analyze the accuracy of the synthetic data in mimicking real student data. Our findings suggest that LLMs can be used to generate synthetic incorrect submissions that are not significantly different from real student data with regard to test case failure distributions. Our research contributes to the development of reliable synthetic datasets for computing education research and teaching, potentially accelerating progress in the field while preserving student privacy. |
| title | LLM-itation is the Sincerest Form of Data: Generating Synthetic Buggy Code Submissions for Computing Education |
| topic | Computers and Society Software Engineering |
| url | https://arxiv.org/abs/2411.10455 |