Automating Thematic Analysis: How LLMs Analyse Controversial Topics
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914951482834944 |
|---|---|
| author | Khan, Awais Hameed Kegalle, Hiruni D'Silva, Rhea Watt, Ned Whelan-Shamy, Daniel Ghahremanlou, Lida Magee, Liam |
| author_facet | Khan, Awais Hameed Kegalle, Hiruni D'Silva, Rhea Watt, Ned Whelan-Shamy, Daniel Ghahremanlou, Lida Magee, Liam |
| contents | Large Language Models (LLMs) are promising analytical tools. They can augment human epistemic, cognitive and reasoning abilities, and support 'sensemaking', making sense of a complex environment or subject by analysing large volumes of data with a sensitivity to context and nuance absent in earlier text processing systems. This paper presents a pilot experiment that explores how LLMs can support thematic analysis of controversial topics. We compare how human researchers and two LLMs GPT-4 and Llama 2 categorise excerpts from media coverage of the controversial Australian Robodebt scandal. Our findings highlight intriguing overlaps and variances in thematic categorisation between human and machine agents, and suggest where LLMs can be effective in supporting forms of discourse and thematic analysis. We argue LLMs should be used to augment, and not replace human interpretation, and we add further methodological insights and reflections to existing research on the application of automation to qualitative research methods. We also introduce a novel card-based design toolkit, for both researchers and practitioners to further interrogate LLMs as analytical tools. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_06919 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Automating Thematic Analysis: How LLMs Analyse Controversial Topics Khan, Awais Hameed Kegalle, Hiruni D'Silva, Rhea Watt, Ned Whelan-Shamy, Daniel Ghahremanlou, Lida Magee, Liam Computers and Society Computation and Language K.4.2 Large Language Models (LLMs) are promising analytical tools. They can augment human epistemic, cognitive and reasoning abilities, and support 'sensemaking', making sense of a complex environment or subject by analysing large volumes of data with a sensitivity to context and nuance absent in earlier text processing systems. This paper presents a pilot experiment that explores how LLMs can support thematic analysis of controversial topics. We compare how human researchers and two LLMs GPT-4 and Llama 2 categorise excerpts from media coverage of the controversial Australian Robodebt scandal. Our findings highlight intriguing overlaps and variances in thematic categorisation between human and machine agents, and suggest where LLMs can be effective in supporting forms of discourse and thematic analysis. We argue LLMs should be used to augment, and not replace human interpretation, and we add further methodological insights and reflections to existing research on the application of automation to qualitative research methods. We also introduce a novel card-based design toolkit, for both researchers and practitioners to further interrogate LLMs as analytical tools. |
| title | Automating Thematic Analysis: How LLMs Analyse Controversial Topics |
| topic | Computers and Society Computation and Language K.4.2 |
| url | https://arxiv.org/abs/2405.06919 |