Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929542899171328 |
|---|---|
| author | Salim, Saiful Islam Yang, Rubin Yuchan Cooper, Alexander Ray, Suryashree Debray, Saumya Rahaman, Sazzadur |
| author_facet | Salim, Saiful Islam Yang, Rubin Yuchan Cooper, Alexander Ray, Suryashree Debray, Saumya Rahaman, Sazzadur |
| contents | While Large language model (LLM)-based programming assistants such as CoPilot and ChatGPT can help improve the productivity of professional software developers, they can also facilitate cheating in introductory computer programming courses. Assuming instructors have limited control over the industrial-strength models, this paper investigates the baseline performance of 5 widely used LLMs on a collection of introductory programming problems, examines adversarial perturbations to degrade their performance, and describes the results of a user study aimed at understanding the efficacy of such perturbations in hindering actual code generation for introductory programming assignments. The user study suggests that i) perturbations combinedly reduced the average correctness score by 77%, ii) the drop in correctness caused by these perturbations was affected based on their detectability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09318 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation Salim, Saiful Islam Yang, Rubin Yuchan Cooper, Alexander Ray, Suryashree Debray, Saumya Rahaman, Sazzadur Computation and Language Computers and Society Software Engineering While Large language model (LLM)-based programming assistants such as CoPilot and ChatGPT can help improve the productivity of professional software developers, they can also facilitate cheating in introductory computer programming courses. Assuming instructors have limited control over the industrial-strength models, this paper investigates the baseline performance of 5 widely used LLMs on a collection of introductory programming problems, examines adversarial perturbations to degrade their performance, and describes the results of a user study aimed at understanding the efficacy of such perturbations in hindering actual code generation for introductory programming assignments. The user study suggests that i) perturbations combinedly reduced the average correctness score by 77%, ii) the drop in correctness caused by these perturbations was affected based on their detectability. |
| title | Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation |
| topic | Computation and Language Computers and Society Software Engineering |
| url | https://arxiv.org/abs/2410.09318 |