Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929626537787392 |
|---|---|
| author | Caraeni, Adriana Scarlatos, Alexander Lan, Andrew |
| author_facet | Caraeni, Adriana Scarlatos, Alexander Lan, Andrew |
| contents | Recent advances in generative artificial intelligence (AI) have shown promise in accurately grading open-ended student responses. However, few prior works have explored grading handwritten responses due to a lack of data and the challenge of combining visual and textual information. In this work, we leverage state-of-the-art multi-modal AI models, in particular GPT-4o, to automatically grade handwritten responses to college-level math exams. Using real student responses to questions in a probability theory exam, we evaluate GPT-4o's alignment with ground-truth scores from human graders using various prompting techniques. We find that while providing rubrics improves alignment, the model's overall accuracy is still too low for real-world settings, showing there is significant room for growth in this task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05231 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams Caraeni, Adriana Scarlatos, Alexander Lan, Andrew Computers and Society Computation and Language Machine Learning Recent advances in generative artificial intelligence (AI) have shown promise in accurately grading open-ended student responses. However, few prior works have explored grading handwritten responses due to a lack of data and the challenge of combining visual and textual information. In this work, we leverage state-of-the-art multi-modal AI models, in particular GPT-4o, to automatically grade handwritten responses to college-level math exams. Using real student responses to questions in a probability theory exam, we evaluate GPT-4o's alignment with ground-truth scores from human graders using various prompting techniques. We find that while providing rubrics improves alignment, the model's overall accuracy is still too low for real-world settings, showing there is significant room for growth in this task. |
| title | Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams |
| topic | Computers and Society Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2411.05231 |