Human Perception of LLM-generated Text Content in Social Media Environments

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Radivojevic, Kristina, Chou, Matthew, Badillo-Urquiola, Karla, Brenner, Paul
Format:	Preprint
Published:	2024
Subjects:	Human-Computer Interaction
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866909310863278080
author	Radivojevic, Kristina Chou, Matthew Badillo-Urquiola, Karla Brenner, Paul
author_facet	Radivojevic, Kristina Chou, Matthew Badillo-Urquiola, Karla Brenner, Paul
contents	Emerging technologies, particularly artificial intelligence (AI), and more specifically Large Language Models (LLMs) have provided malicious actors with powerful tools for manipulating digital discourse. LLMs have the potential to affect traditional forms of democratic engagements, such as voter choice, government surveys, or even online communication with regulators; since bots are capable of producing large quantities of credible text. To investigate the human perception of LLM-generated content, we recruited over 1,000 participants who then tried to differentiate bot from human posts in social media discussion threads. We found that humans perform poorly at identifying the true nature of user posts on social media. We also found patterns in how humans identify LLM-generated text content in social media discourse. Finally, we observed the Uncanny Valley effect in text dialogue in both user perception and identification. This indicates that despite humans being poor at the identification process, they can still sense discomfort when reading LLM-generated content.
format	Preprint
id	arxiv_https___arxiv_org_abs_2409_06653
institution	arXiv
publishDate	2024
record_format	arxiv
spellingShingle	Human Perception of LLM-generated Text Content in Social Media Environments Radivojevic, Kristina Chou, Matthew Badillo-Urquiola, Karla Brenner, Paul Human-Computer Interaction Emerging technologies, particularly artificial intelligence (AI), and more specifically Large Language Models (LLMs) have provided malicious actors with powerful tools for manipulating digital discourse. LLMs have the potential to affect traditional forms of democratic engagements, such as voter choice, government surveys, or even online communication with regulators; since bots are capable of producing large quantities of credible text. To investigate the human perception of LLM-generated content, we recruited over 1,000 participants who then tried to differentiate bot from human posts in social media discussion threads. We found that humans perform poorly at identifying the true nature of user posts on social media. We also found patterns in how humans identify LLM-generated text content in social media discourse. Finally, we observed the Uncanny Valley effect in text dialogue in both user perception and identification. This indicates that despite humans being poor at the identification process, they can still sense discomfort when reading LLM-generated content.
title	Human Perception of LLM-generated Text Content in Social Media Environments
topic	Human-Computer Interaction
url	https://arxiv.org/abs/2409.06653