SocialBench: Sociality Evaluation of Role-Playing Conversational Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914898539184128 |
|---|---|
| author | Chen, Hongzhan Chen, Hehong Yan, Ming Xu, Wenshen Gao, Xing Shen, Weizhou Quan, Xiaojun Li, Chenliang Zhang, Ji Huang, Fei Zhou, Jingren |
| author_facet | Chen, Hongzhan Chen, Hehong Yan, Ming Xu, Wenshen Gao, Xing Shen, Weizhou Quan, Xiaojun Li, Chenliang Zhang, Ji Huang, Fei Zhou, Jingren |
| contents | Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/SocialBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13679 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SocialBench: Sociality Evaluation of Role-Playing Conversational Agents Chen, Hongzhan Chen, Hehong Yan, Ming Xu, Wenshen Gao, Xing Shen, Weizhou Quan, Xiaojun Li, Chenliang Zhang, Ji Huang, Fei Zhou, Jingren Computation and Language Large language models (LLMs) have advanced the development of various AI conversational agents, including role-playing conversational agents that mimic diverse characters and human behaviors. While prior research has predominantly focused on enhancing the conversational capability, role-specific knowledge, and stylistic attributes of these agents, there has been a noticeable gap in assessing their social intelligence. In this paper, we introduce SocialBench, the first benchmark designed to systematically evaluate the sociality of role-playing conversational agents at both individual and group levels of social interactions. The benchmark is constructed from a variety of sources and covers a wide range of 500 characters and over 6,000 question prompts and 30,800 multi-turn role-playing utterances. We conduct comprehensive evaluations on this benchmark using mainstream open-source and closed-source LLMs. We find that agents excelling in individual level does not imply their proficiency in group level. Moreover, the behavior of individuals may drift as a result of the influence exerted by other agents within the group. Experimental results on SocialBench confirm its significance as a testbed for assessing the social interaction of role-playing conversational agents. The benchmark is publicly accessible at https://github.com/X-PLUG/SocialBench. |
| title | SocialBench: Sociality Evaluation of Role-Playing Conversational Agents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.13679 |