Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910606515240960 |
|---|---|
| author | Ivey, Jonathan Kumar, Shivani Liu, Jiayu Shen, Hua Rakshit, Sushrita Raju, Rohan Zhang, Haotian Ananthasubramaniam, Aparna Kim, Junghwan Yi, Bowen Wright, Dustin Israeli, Abraham Møller, Anders Giovanni Zhang, Lechen Jurgens, David |
| author_facet | Ivey, Jonathan Kumar, Shivani Liu, Jiayu Shen, Hua Rakshit, Sushrita Raju, Rohan Zhang, Haotian Ananthasubramaniam, Aparna Kim, Junghwan Yi, Bowen Wright, Dustin Israeli, Abraham Møller, Anders Giovanni Zhang, Lechen Jurgens, David |
| contents | Studying and building datasets for dialogue tasks is both expensive and time-consuming due to the need to recruit, train, and collect data from study participants. In response, much recent work has sought to use large language models (LLMs) to simulate both human-human and human-LLM interactions, as they have been shown to generate convincingly human-like text in many settings. However, to what extent do LLM-based simulations \textit{actually} reflect human dialogues? In this work, we answer this question by generating a large-scale dataset of 100,000 paired LLM-LLM and human-LLM dialogues from the WildChat dataset and quantifying how well the LLM simulations align with their human counterparts. Overall, we find relatively low alignment between simulations and human interactions, demonstrating a systematic divergence along the multiple textual properties, including style and content. Further, in comparisons of English, Chinese, and Russian dialogues, we find that models perform similarly. Our results suggest that LLMs generally perform better when the human themself writes in a way that is more similar to the LLM's own style. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08330 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue Ivey, Jonathan Kumar, Shivani Liu, Jiayu Shen, Hua Rakshit, Sushrita Raju, Rohan Zhang, Haotian Ananthasubramaniam, Aparna Kim, Junghwan Yi, Bowen Wright, Dustin Israeli, Abraham Møller, Anders Giovanni Zhang, Lechen Jurgens, David Computation and Language Computers and Society Human-Computer Interaction Studying and building datasets for dialogue tasks is both expensive and time-consuming due to the need to recruit, train, and collect data from study participants. In response, much recent work has sought to use large language models (LLMs) to simulate both human-human and human-LLM interactions, as they have been shown to generate convincingly human-like text in many settings. However, to what extent do LLM-based simulations \textit{actually} reflect human dialogues? In this work, we answer this question by generating a large-scale dataset of 100,000 paired LLM-LLM and human-LLM dialogues from the WildChat dataset and quantifying how well the LLM simulations align with their human counterparts. Overall, we find relatively low alignment between simulations and human interactions, demonstrating a systematic divergence along the multiple textual properties, including style and content. Further, in comparisons of English, Chinese, and Russian dialogues, we find that models perform similarly. Our results suggest that LLMs generally perform better when the human themself writes in a way that is more similar to the LLM's own style. |
| title | Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue |
| topic | Computation and Language Computers and Society Human-Computer Interaction |
| url | https://arxiv.org/abs/2409.08330 |