Scaling up the think-aloud method
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913866588356608 |
|---|---|
| author | Wurgaft, Daniel Prystawski, Ben Gandhi, Kanishk Zhang, Cedegao E. Tenenbaum, Joshua B. Goodman, Noah D. |
| author_facet | Wurgaft, Daniel Prystawski, Ben Gandhi, Kanishk Zhang, Cedegao E. Tenenbaum, Joshua B. Goodman, Noah D. |
| contents | The think-aloud method, where participants voice their thoughts as they solve a task, is a valuable source of rich data about human reasoning processes. Yet, it has declined in popularity in contemporary cognitive science, largely because labor-intensive transcription and annotation preclude large sample sizes. Here, we develop methods to automate the transcription and annotation of verbal reports of reasoning using natural language processing tools, allowing for large-scale analysis of think-aloud data. In our study, 640 participants thought aloud while playing the Game of 24, a mathematical reasoning task. We automatically transcribed the recordings and coded the transcripts as search graphs, finding moderate inter-rater reliability with humans. We analyze these graphs and characterize consistency and variation in human reasoning traces. Our work demonstrates the value of think-aloud data at scale and serves as a proof of concept for the automated analysis of verbal reports. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23931 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling up the think-aloud method Wurgaft, Daniel Prystawski, Ben Gandhi, Kanishk Zhang, Cedegao E. Tenenbaum, Joshua B. Goodman, Noah D. Computation and Language Artificial Intelligence The think-aloud method, where participants voice their thoughts as they solve a task, is a valuable source of rich data about human reasoning processes. Yet, it has declined in popularity in contemporary cognitive science, largely because labor-intensive transcription and annotation preclude large sample sizes. Here, we develop methods to automate the transcription and annotation of verbal reports of reasoning using natural language processing tools, allowing for large-scale analysis of think-aloud data. In our study, 640 participants thought aloud while playing the Game of 24, a mathematical reasoning task. We automatically transcribed the recordings and coded the transcripts as search graphs, finding moderate inter-rater reliability with humans. We analyze these graphs and characterize consistency and variation in human reasoning traces. Our work demonstrates the value of think-aloud data at scale and serves as a proof of concept for the automated analysis of verbal reports. |
| title | Scaling up the think-aloud method |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.23931 |