RACER: An LLM-powered Methodology for Scalable Analysis of Semi-structured Mental Health Interviews
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929234237194240 |
|---|---|
| author | Singh, Satpreet Harcharan Jiang, Kevin Bhasin, Kanchan Sabharwal, Ashutosh Moukaddam, Nidal Patel, Ankit B |
| author_facet | Singh, Satpreet Harcharan Jiang, Kevin Bhasin, Kanchan Sabharwal, Ashutosh Moukaddam, Nidal Patel, Ankit B |
| contents | Semi-structured interviews (SSIs) are a commonly employed data-collection method in healthcare research, offering in-depth qualitative insights into subject experiences. Despite their value, the manual analysis of SSIs is notoriously time-consuming and labor-intensive, in part due to the difficulty of extracting and categorizing emotional responses, and challenges in scaling human evaluation for large populations. In this study, we develop RACER, a Large Language Model (LLM) based expert-guided automated pipeline that efficiently converts raw interview transcripts into insightful domain-relevant themes and sub-themes. We used RACER to analyze SSIs conducted with 93 healthcare professionals and trainees to assess the broad personal and professional mental health impacts of the COVID-19 crisis. RACER achieves moderately high agreement with two human evaluators (72%), which approaches the human inter-rater agreement (77%). Interestingly, LLMs and humans struggle with similar content involving nuanced emotional, ambivalent/dialectical, and psychological statements. Our study highlights the opportunities and challenges in using LLMs to improve research efficiency and opens new avenues for scalable analysis of SSIs in healthcare research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02656 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RACER: An LLM-powered Methodology for Scalable Analysis of Semi-structured Mental Health Interviews Singh, Satpreet Harcharan Jiang, Kevin Bhasin, Kanchan Sabharwal, Ashutosh Moukaddam, Nidal Patel, Ankit B Computation and Language Quantitative Methods Semi-structured interviews (SSIs) are a commonly employed data-collection method in healthcare research, offering in-depth qualitative insights into subject experiences. Despite their value, the manual analysis of SSIs is notoriously time-consuming and labor-intensive, in part due to the difficulty of extracting and categorizing emotional responses, and challenges in scaling human evaluation for large populations. In this study, we develop RACER, a Large Language Model (LLM) based expert-guided automated pipeline that efficiently converts raw interview transcripts into insightful domain-relevant themes and sub-themes. We used RACER to analyze SSIs conducted with 93 healthcare professionals and trainees to assess the broad personal and professional mental health impacts of the COVID-19 crisis. RACER achieves moderately high agreement with two human evaluators (72%), which approaches the human inter-rater agreement (77%). Interestingly, LLMs and humans struggle with similar content involving nuanced emotional, ambivalent/dialectical, and psychological statements. Our study highlights the opportunities and challenges in using LLMs to improve research efficiency and opens new avenues for scalable analysis of SSIs in healthcare research. |
| title | RACER: An LLM-powered Methodology for Scalable Analysis of Semi-structured Mental Health Interviews |
| topic | Computation and Language Quantitative Methods |
| url | https://arxiv.org/abs/2402.02656 |