Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.00775 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914115645079552 |
|---|---|
| author | Gao, Jie Shu, Zhiyao Yeo, Shun Yi Prakash, Alok Huang, Chien-Ming Dredze, Mark Xiao, Ziang |
| author_facet | Gao, Jie Shu, Zhiyao Yeo, Shun Yi Prakash, Alok Huang, Chien-Ming Dredze, Mark Xiao, Ziang |
| contents | Qualitative data analysis (QDA) emphasizes trustworthiness, requiring sustained human engagement and reflexivity. Recently, large language models (LLMs) have been applied in QDA to improve efficiency. However, their use raises concerns about unvalidated automation and displaced sensemaking, which can undermine trustworthiness. To address these issues, we employed two strategies: transparency and human involvement. Through a literature review and formative interviews, we identified six design requirements for transparent automation and meaningful human involvement. Guided by these requirements, we developed MindCoder, an LLM-powered workflow that delegates mechanical tasks, such as grouping and validation, to the system, while enabling humans to conduct meaningful interpretation. MindCoder also maintains comprehensive logs of users' step-by-step interactions to ensure transparency and support trustworthy results. In an evaluation with 12 users and two external evaluators, MindCoder supported active interpretation, offered flexible control, and produced more trustworthy codebooks. We further discuss design implications for building human-AI collaborative QDA workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00775 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficiency with Rigor! A Trustworthy LLM-powered Workflow for Qualitative Data Analysis Gao, Jie Shu, Zhiyao Yeo, Shun Yi Prakash, Alok Huang, Chien-Ming Dredze, Mark Xiao, Ziang Human-Computer Interaction Qualitative data analysis (QDA) emphasizes trustworthiness, requiring sustained human engagement and reflexivity. Recently, large language models (LLMs) have been applied in QDA to improve efficiency. However, their use raises concerns about unvalidated automation and displaced sensemaking, which can undermine trustworthiness. To address these issues, we employed two strategies: transparency and human involvement. Through a literature review and formative interviews, we identified six design requirements for transparent automation and meaningful human involvement. Guided by these requirements, we developed MindCoder, an LLM-powered workflow that delegates mechanical tasks, such as grouping and validation, to the system, while enabling humans to conduct meaningful interpretation. MindCoder also maintains comprehensive logs of users' step-by-step interactions to ensure transparency and support trustworthy results. In an evaluation with 12 users and two external evaluators, MindCoder supported active interpretation, offered flexible control, and produced more trustworthy codebooks. We further discuss design implications for building human-AI collaborative QDA workflows. |
| title | Efficiency with Rigor! A Trustworthy LLM-powered Workflow for Qualitative Data Analysis |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2501.00775 |