Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2412.01312 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917853463052288 |
|---|---|
| author | Pimbblet, Kevin A. Morrell, Lesley J. |
| author_facet | Pimbblet, Kevin A. Morrell, Lesley J. |
| contents | The emergence of conversational natural language processing models presents a significant challenge for Higher Education. In this work, we use the entirety of a UK physics undergraduate (BSc with Honours) degree including all examinations and coursework to test if ChatGPT (GPT-4) can pass a degree. We adopt a "maximal cheating" approach wherein we permit ourselves to modify questions for clarity, split questions up into smaller sub-components, expand on answers given - especially for long form written responses, obtaining references, and use of advanced coaching, plug-ins and custom instructions to optimize outputs. In general, there are only certain parts of the degree in question where GPT-4 fails. Explicitly these include compulsory laboratory elements, and the final project which is assessed by a viva. If these were no issue, then GPT-4 would pass with a grade of an upper second class overall. In general, coding tasks are performed exceptionally well, along with simple single-step solution problems. Multiple step problems and longer prose are generally poorer along with interdisciplinary problems. We strongly suggest that there is now a necessity to urgently re-think and revise assessment practice in physics - and other disciplines - due to the existence of AI such as GPT-4. We recommend close scrutiny of assessment tasks: only invigilated in-person examinations, vivas, laboratory skills testing (or "performances" in other disciplines), and presentations are not vulnerable to GPT-4, and urge consideration of how AI can be embedded within the disciplinary context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01312 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degrees Pimbblet, Kevin A. Morrell, Lesley J. Physics Education The emergence of conversational natural language processing models presents a significant challenge for Higher Education. In this work, we use the entirety of a UK physics undergraduate (BSc with Honours) degree including all examinations and coursework to test if ChatGPT (GPT-4) can pass a degree. We adopt a "maximal cheating" approach wherein we permit ourselves to modify questions for clarity, split questions up into smaller sub-components, expand on answers given - especially for long form written responses, obtaining references, and use of advanced coaching, plug-ins and custom instructions to optimize outputs. In general, there are only certain parts of the degree in question where GPT-4 fails. Explicitly these include compulsory laboratory elements, and the final project which is assessed by a viva. If these were no issue, then GPT-4 would pass with a grade of an upper second class overall. In general, coding tasks are performed exceptionally well, along with simple single-step solution problems. Multiple step problems and longer prose are generally poorer along with interdisciplinary problems. We strongly suggest that there is now a necessity to urgently re-think and revise assessment practice in physics - and other disciplines - due to the existence of AI such as GPT-4. We recommend close scrutiny of assessment tasks: only invigilated in-person examinations, vivas, laboratory skills testing (or "performances" in other disciplines), and presentations are not vulnerable to GPT-4, and urge consideration of how AI can be embedded within the disciplinary context. |
| title | Can ChatGPT pass a physics degree? Making a case for reformation of assessment of undergraduate degrees |
| topic | Physics Education |
| url | https://arxiv.org/abs/2412.01312 |