The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916425234382848 |
|---|---|
| author | Choi, Alexander S. Akter, Syeda Sabrina Singh, JP Anastasopoulos, Antonios |
| author_facet | Choi, Alexander S. Akter, Syeda Sabrina Singh, JP Anastasopoulos, Antonios |
| contents | Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks, leading researchers to use them for time and labor-intensive analyses. However, their capability to handle highly specialized and open-ended tasks in domains like policy studies remains in question. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership. The study, conducted in two stages-Topic Discovery and Topic Assignment-integrates LLMs with expert annotators to observe the impact of LLM suggestions on what is usually human-only analysis. Results indicate that LLM-generated topic lists have significant overlap with human generated topic lists, with minor hiccups in missing document-specific topics. However, LLM suggestions may significantly improve task completion speed, but at the same time introduce anchoring bias, potentially affecting the depth and nuance of the analysis, raising a critical question about the trade-off between increased efficiency and the risk of biased analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04699 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? Choi, Alexander S. Akter, Syeda Sabrina Singh, JP Anastasopoulos, Antonios Computation and Language Human-Computer Interaction Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks, leading researchers to use them for time and labor-intensive analyses. However, their capability to handle highly specialized and open-ended tasks in domains like policy studies remains in question. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership. The study, conducted in two stages-Topic Discovery and Topic Assignment-integrates LLMs with expert annotators to observe the impact of LLM suggestions on what is usually human-only analysis. Results indicate that LLM-generated topic lists have significant overlap with human generated topic lists, with minor hiccups in missing document-specific topics. However, LLM suggestions may significantly improve task completion speed, but at the same time introduce anchoring bias, potentially affecting the depth and nuance of the analysis, raising a critical question about the trade-off between increased efficiency and the risk of biased analysis. |
| title | The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? |
| topic | Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.04699 |