'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913522470879232 |
|---|---|
| author | Kapania, Shivani Agnew, William Eslami, Motahhare Heidari, Hoda Fox, Sarah |
| author_facet | Kapania, Shivani Agnew, William Eslami, Motahhare Heidari, Hoda Fox, Sarah |
| contents | The recent excitement around generative models has sparked a wave of proposals suggesting the replacement of human participation and labor in research and development--e.g., through surveys, experiments, and interviews--with synthetic research data generated by large language models (LLMs). We conducted interviews with 19 qualitative researchers to understand their perspectives on this paradigm shift. Initially skeptical, researchers were surprised to see similar narratives emerge in the LLM-generated data when using the interview probe. However, over several conversational turns, they went on to identify fundamental limitations, such as how LLMs foreclose participants' consent and agency, produce responses lacking in palpability and contextual depth, and risk delegitimizing qualitative research methods. We argue that the use of LLMs as proxies for participants enacts the surrogate effect, raising ethical and epistemological concerns that extend beyond the technical limitations of current models to the core of whether LLMs fit within qualitative ways of knowing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_19430 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | 'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants Kapania, Shivani Agnew, William Eslami, Motahhare Heidari, Hoda Fox, Sarah Human-Computer Interaction Computation and Language Machine Learning The recent excitement around generative models has sparked a wave of proposals suggesting the replacement of human participation and labor in research and development--e.g., through surveys, experiments, and interviews--with synthetic research data generated by large language models (LLMs). We conducted interviews with 19 qualitative researchers to understand their perspectives on this paradigm shift. Initially skeptical, researchers were surprised to see similar narratives emerge in the LLM-generated data when using the interview probe. However, over several conversational turns, they went on to identify fundamental limitations, such as how LLMs foreclose participants' consent and agency, produce responses lacking in palpability and contextual depth, and risk delegitimizing qualitative research methods. We argue that the use of LLMs as proxies for participants enacts the surrogate effect, raising ethical and epistemological concerns that extend beyond the technical limitations of current models to the core of whether LLMs fit within qualitative ways of knowing. |
| title | 'Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants |
| topic | Human-Computer Interaction Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2409.19430 |