Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918028772376576 |
|---|---|
| author | Zhang, Bang Ma, Ruotian Jiang, Qingxuan Wang, Peisong Chen, Jiaqi Xie, Zheng Chen, Xingyu Wang, Yue Ye, Fanghua Li, Jian Yang, Yifan Tu, Zhaopeng Li, Xiaolong |
| author_facet | Zhang, Bang Ma, Ruotian Jiang, Qingxuan Wang, Peisong Chen, Jiaqi Xie, Zheng Chen, Xingyu Wang, Yue Ye, Fanghua Li, Jian Yang, Yifan Tu, Zhaopeng Li, Xiaolong |
| contents | Assessing how well a large language model (LLM) understands human, rather than merely text, remains an open challenge. To bridge the gap, we introduce Sentient Agent as a Judge (SAGE), an automated evaluation framework that measures an LLM's higher-order social cognition. SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction, providing a more realistic evaluation of the tested model in multi-turn conversations. At every turn, the agent reasons about (i) how its emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a numerical emotion trajectory and interpretable inner thoughts. Experiments on 100 supportive-dialogue scenarios show that the final Sentient emotion score correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings and utterance-level empathy metrics, validating psychological fidelity. We also build a public Sentient Leaderboard covering 18 commercial and open-source models that uncovers substantial gaps (up to 4x) between frontier systems (GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in conventional leaderboards (e.g., Arena). SAGE thus provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02847 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models Zhang, Bang Ma, Ruotian Jiang, Qingxuan Wang, Peisong Chen, Jiaqi Xie, Zheng Chen, Xingyu Wang, Yue Ye, Fanghua Li, Jian Yang, Yifan Tu, Zhaopeng Li, Xiaolong Computation and Language Artificial Intelligence Computers and Society Assessing how well a large language model (LLM) understands human, rather than merely text, remains an open challenge. To bridge the gap, we introduce Sentient Agent as a Judge (SAGE), an automated evaluation framework that measures an LLM's higher-order social cognition. SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction, providing a more realistic evaluation of the tested model in multi-turn conversations. At every turn, the agent reasons about (i) how its emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a numerical emotion trajectory and interpretable inner thoughts. Experiments on 100 supportive-dialogue scenarios show that the final Sentient emotion score correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings and utterance-level empathy metrics, validating psychological fidelity. We also build a public Sentient Leaderboard covering 18 commercial and open-source models that uncovers substantial gaps (up to 4x) between frontier systems (GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in conventional leaderboards (e.g., Arena). SAGE thus provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents. |
| title | Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models |
| topic | Computation and Language Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2505.02847 |