Risk or Chance? Large Language Models and Reproducibility in HCI Research
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929570888810496 |
|---|---|
| author | Kosch, Thomas Feger, Sebastian |
| author_facet | Kosch, Thomas Feger, Sebastian |
| contents | Reproducibility is a major concern across scientific fields. Human-Computer Interaction (HCI), in particular, is subject to diverse reproducibility challenges due to the wide range of research methodologies employed. In this article, we explore how the increasing adoption of Large Language Models (LLMs) across all user experience (UX) design and research activities impacts reproducibility in HCI. In particular, we review upcoming reproducibility challenges through the lenses of analogies from past to future (mis)practices like p-hacking and prompt-hacking, general bias, support in data analysis, documentation and education requirements, and possible pressure on the community. We discuss the risks and chances for each of these lenses with the expectation that a more comprehensive discussion will help shape best practices and contribute to valid and reproducible practices around using LLMs in HCI research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_15782 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Risk or Chance? Large Language Models and Reproducibility in HCI Research Kosch, Thomas Feger, Sebastian Human-Computer Interaction Reproducibility is a major concern across scientific fields. Human-Computer Interaction (HCI), in particular, is subject to diverse reproducibility challenges due to the wide range of research methodologies employed. In this article, we explore how the increasing adoption of Large Language Models (LLMs) across all user experience (UX) design and research activities impacts reproducibility in HCI. In particular, we review upcoming reproducibility challenges through the lenses of analogies from past to future (mis)practices like p-hacking and prompt-hacking, general bias, support in data analysis, documentation and education requirements, and possible pressure on the community. We discuss the risks and chances for each of these lenses with the expectation that a more comprehensive discussion will help shape best practices and contribute to valid and reproducible practices around using LLMs in HCI research. |
| title | Risk or Chance? Large Language Models and Reproducibility in HCI Research |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2404.15782 |