Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914094671462400 |
|---|---|
| author | Hairston, JaMor Ranjan, Ritvik Lakamana, Sahithi Spadaro, Anthony Bozkurt, Selen Perrone, Jeanmarie Sarker, Abeed |
| author_facet | Hairston, JaMor Ranjan, Ritvik Lakamana, Sahithi Spadaro, Anthony Bozkurt, Selen Perrone, Jeanmarie Sarker, Abeed |
| contents | Background Large language models (LLMs) face challenges in inductive thematic analysis, a task requiring deep interpretive and domain-specific expertise. We evaluated the feasibility of using LLMs to replicate expert-driven thematic analysis of social media data. Methods Using two temporally non-intersecting Reddit datasets on xylazine (n=286 and n=686, for model optimization and validation, respectively) with twelve expert-derived themes, we evaluated five LLMs against expert coding. We modeled the task as a series of binary classifications, rather than a single, multi-label classification, employing zero-, single-, and few-shot prompting strategies and measuring performance via accuracy, precision, recall, and F1-score. Results On the validation set, GPT-4o with two-shot prompting performed best (accuracy: 90.9%; F1-score: 0.71). For high-prevalence themes, model-derived thematic distributions closely mirrored expert classifications (e.g., xylazine use: 13.6% vs. 17.8%; MOUD use: 16.5% vs. 17.8%). Conclusions Our findings suggest that few-shot LLM-based approaches can automate thematic analyses, offering a scalable supplement for qualitative research. Keywords: thematic analysis, large language models, natural language processing, qualitative analysis, social media, prompt engineering, public health |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_10803 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case Hairston, JaMor Ranjan, Ritvik Lakamana, Sahithi Spadaro, Anthony Bozkurt, Selen Perrone, Jeanmarie Sarker, Abeed Artificial Intelligence Computation and Language Emerging Technologies Information Retrieval Background Large language models (LLMs) face challenges in inductive thematic analysis, a task requiring deep interpretive and domain-specific expertise. We evaluated the feasibility of using LLMs to replicate expert-driven thematic analysis of social media data. Methods Using two temporally non-intersecting Reddit datasets on xylazine (n=286 and n=686, for model optimization and validation, respectively) with twelve expert-derived themes, we evaluated five LLMs against expert coding. We modeled the task as a series of binary classifications, rather than a single, multi-label classification, employing zero-, single-, and few-shot prompting strategies and measuring performance via accuracy, precision, recall, and F1-score. Results On the validation set, GPT-4o with two-shot prompting performed best (accuracy: 90.9%; F1-score: 0.71). For high-prevalence themes, model-derived thematic distributions closely mirrored expert classifications (e.g., xylazine use: 13.6% vs. 17.8%; MOUD use: 16.5% vs. 17.8%). Conclusions Our findings suggest that few-shot LLM-based approaches can automate thematic analyses, offering a scalable supplement for qualitative research. Keywords: thematic analysis, large language models, natural language processing, qualitative analysis, social media, prompt engineering, public health |
| title | Automated Thematic Analyses Using LLMs: Xylazine Wound Management Social Media Chatter Use Case |
| topic | Artificial Intelligence Computation and Language Emerging Technologies Information Retrieval |
| url | https://arxiv.org/abs/2507.10803 |