Processes Matter: How ML/GAI Approaches Could Support Open Qualitative Coding of Online Discourse Datasets
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916673481605120 |
|---|---|
| author | Chen, John Lotsos, Alexandros Wang, Grace Zhao, Lexie Sherin, Bruce Wilensky, Uri Horn, Michael |
| author_facet | Chen, John Lotsos, Alexandros Wang, Grace Zhao, Lexie Sherin, Bruce Wilensky, Uri Horn, Michael |
| contents | Open coding, a key inductive step in qualitative research, discovers and constructs concepts from human datasets. However, capturing extensive and nuanced aspects or "coding moments" can be challenging, especially with large discourse datasets. While some studies explore machine learning (ML)/Generative AI (GAI)'s potential for open coding, few evaluation studies exist. We compare open coding results by five recently published ML/GAI approaches and four human coders, using a dataset of online chat messages around a mobile learning software. Our systematic analysis reveals ML/GAI approaches' strengths and weaknesses, uncovering the complementary potential between humans and AI. Line-by-line AI approaches effectively identify content-based codes, while humans excel in interpreting conversational dynamics. We discussed how embedded analytical processes could shape the results of ML/GAI approaches. Instead of replacing humans in open coding, researchers should integrate AI with and according to their analytical processes, e.g., as parallel co-coders. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02887 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Processes Matter: How ML/GAI Approaches Could Support Open Qualitative Coding of Online Discourse Datasets Chen, John Lotsos, Alexandros Wang, Grace Zhao, Lexie Sherin, Bruce Wilensky, Uri Horn, Michael Computation and Language Human-Computer Interaction Machine Learning Open coding, a key inductive step in qualitative research, discovers and constructs concepts from human datasets. However, capturing extensive and nuanced aspects or "coding moments" can be challenging, especially with large discourse datasets. While some studies explore machine learning (ML)/Generative AI (GAI)'s potential for open coding, few evaluation studies exist. We compare open coding results by five recently published ML/GAI approaches and four human coders, using a dataset of online chat messages around a mobile learning software. Our systematic analysis reveals ML/GAI approaches' strengths and weaknesses, uncovering the complementary potential between humans and AI. Line-by-line AI approaches effectively identify content-based codes, while humans excel in interpreting conversational dynamics. We discussed how embedded analytical processes could shape the results of ML/GAI approaches. Instead of replacing humans in open coding, researchers should integrate AI with and according to their analytical processes, e.g., as parallel co-coders. |
| title | Processes Matter: How ML/GAI Approaches Could Support Open Qualitative Coding of Online Discourse Datasets |
| topic | Computation and Language Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2504.02887 |