Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.10246 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918366873124864 |
|---|---|
| author | Hao, Jiangang Cui, Wenju Kyllonen, Patrick Kerzabi, Emily Liu, Lei Flor, Michael |
| author_facet | Hao, Jiangang Cui, Wenju Kyllonen, Patrick Kerzabi, Emily Liu, Lei Flor, Michael |
| contents | Collaborative problem solving (CPS) is widely recognized as a critical 21st-century skill. Assessing CPS depends heavily on coding the communication data using a construct-relevant framework, and this process has long been a major bottleneck to scaling up such assessments. Based on five datasets and two coding frameworks, we demonstrate that ChatGPT can code communication data to a satisfactory level, though performance varies across ChatGPT models, and depends on the coding framework and task characteristics. Interestingly, newer reasoning-focused models such as GPT-o1-mini and GPT-o3-mini do not necessarily yield better coding results. Additionally, we show that refining prompts based on feedback from miscoded cases can improve coding accuracy in some instances, though the effectiveness of this approach is not consistent across all tasks. These findings offer practical guidance for researchers and practitioners in developing scalable, efficient methods to analyze communication data in support of 21st-century skill assessment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10246 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Automated Coding of Communications in Collaborative Problem-solving Tasks Using ChatGPT Hao, Jiangang Cui, Wenju Kyllonen, Patrick Kerzabi, Emily Liu, Lei Flor, Michael Human-Computer Interaction Computation and Language Collaborative problem solving (CPS) is widely recognized as a critical 21st-century skill. Assessing CPS depends heavily on coding the communication data using a construct-relevant framework, and this process has long been a major bottleneck to scaling up such assessments. Based on five datasets and two coding frameworks, we demonstrate that ChatGPT can code communication data to a satisfactory level, though performance varies across ChatGPT models, and depends on the coding framework and task characteristics. Interestingly, newer reasoning-focused models such as GPT-o1-mini and GPT-o3-mini do not necessarily yield better coding results. Additionally, we show that refining prompts based on feedback from miscoded cases can improve coding accuracy in some instances, though the effectiveness of this approach is not consistent across all tasks. These findings offer practical guidance for researchers and practitioners in developing scalable, efficient methods to analyze communication data in support of 21st-century skill assessment. |
| title | Automated Coding of Communications in Collaborative Problem-solving Tasks Using ChatGPT |
| topic | Human-Computer Interaction Computation and Language |
| url | https://arxiv.org/abs/2411.10246 |