Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917247277072384 |
|---|---|
| author | Kim, Yoonsu Jin, Hyoungwook Doh, Hayeon Kim, Eunhye Jung, Dongyun Kim, Seungju Choi, Kiyoon Son, Jinho Kim, Juho |
| author_facet | Kim, Yoonsu Jin, Hyoungwook Doh, Hayeon Kim, Eunhye Jung, Dongyun Kim, Seungju Choi, Kiyoon Son, Jinho Kim, Juho |
| contents | Students' handwritten math work provides a rich resource for diagnosing cognitive skills, as it captures intermediate reasoning beyond final answers. We investigate how current large language models (LLMs) perform in diagnosing cognitive skills from such work. However, student responses vary widely, often omitting steps or providing only vague, contextually implicit evidence. Despite recent advances in LLMs' multimodal and reasoning capabilities, their performance under such conditions remains underexplored. To address this gap, we constructed MathCog, a benchmark dataset containing 3,036 diagnostic verdicts across 639 student responses to 110 math problems, annotated by teachers using TIMSS-grounded cognitive skill checklists with evidential strength labels (Evident/Vague). Evaluating 18 LLMs, we find that (1) all models underperform (F1 < 0.5) regardless of capability, and (2) performance degrades sharply under vague evidence. Error analysis reveals systematic patterns: models frequently misattribute Vague evidence as Evident, overthink minimal cues, and hallucinate nonexistent evidence. We discuss implications for evidence-aware, teacher-in-the-loop designs for LLM-based cognitive diagnosis in educational settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_00843 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work Kim, Yoonsu Jin, Hyoungwook Doh, Hayeon Kim, Eunhye Jung, Dongyun Kim, Seungju Choi, Kiyoon Son, Jinho Kim, Juho Artificial Intelligence Human-Computer Interaction Students' handwritten math work provides a rich resource for diagnosing cognitive skills, as it captures intermediate reasoning beyond final answers. We investigate how current large language models (LLMs) perform in diagnosing cognitive skills from such work. However, student responses vary widely, often omitting steps or providing only vague, contextually implicit evidence. Despite recent advances in LLMs' multimodal and reasoning capabilities, their performance under such conditions remains underexplored. To address this gap, we constructed MathCog, a benchmark dataset containing 3,036 diagnostic verdicts across 639 student responses to 110 math problems, annotated by teachers using TIMSS-grounded cognitive skill checklists with evidential strength labels (Evident/Vague). Evaluating 18 LLMs, we find that (1) all models underperform (F1 < 0.5) regardless of capability, and (2) performance degrades sharply under vague evidence. Error analysis reveals systematic patterns: models frequently misattribute Vague evidence as Evident, overthink minimal cues, and hallucinate nonexistent evidence. We discuss implications for evidence-aware, teacher-in-the-loop designs for LLM-based cognitive diagnosis in educational settings. |
| title | Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work |
| topic | Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2504.00843 |