Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.08592 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912994135375872 |
|---|---|
| author | Bouleimen, Azza De Marzo, Giordano Kim, Taehee Pagan, Nicol`o Metzler, Hannah Giordano, Silvia Hannák, Anikó Garcia, David |
| author_facet | Bouleimen, Azza De Marzo, Giordano Kim, Taehee Pagan, Nicol`o Metzler, Hannah Giordano, Silvia Hannák, Anikó Garcia, David |
| contents | Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39\% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56\% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08592 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions Bouleimen, Azza De Marzo, Giordano Kim, Taehee Pagan, Nicol`o Metzler, Hannah Giordano, Silvia Hannák, Anikó Garcia, David Computation and Language Artificial Intelligence Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39\% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56\% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content. |
| title | The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2511.08592 |