ToMBench: Benchmarking Theory of Mind in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929619012157440 |
|---|---|
| author | Chen, Zhuang Wu, Jincenzi Zhou, Jinfeng Wen, Bosi Bi, Guanqun Jiang, Gongyao Cao, Yaru Hu, Mengting Lai, Yunghwei Xiong, Zexuan Huang, Minlie |
| author_facet | Chen, Zhuang Wu, Jincenzi Zhou, Jinfeng Wen, Bosi Bi, Guanqun Jiang, Gongyao Cao, Yaru Hu, Mengting Lai, Yunghwei Xiong, Zexuan Huang, Minlie |
| contents | Theory of Mind (ToM) is the cognitive capability to perceive and ascribe mental states to oneself and others. Recent research has sparked a debate over whether large language models (LLMs) exhibit a form of ToM. However, existing ToM evaluations are hindered by challenges such as constrained scope, subjective judgment, and unintended contamination, yielding inadequate assessments. To address this gap, we introduce ToMBench with three key characteristics: a systematic evaluation framework encompassing 8 tasks and 31 abilities in social cognition, a multiple-choice question format to support automated and unbiased evaluation, and a build-from-scratch bilingual inventory to strictly avoid data leakage. Based on ToMBench, we conduct extensive experiments to evaluate the ToM performance of 10 popular LLMs across tasks and abilities. We find that even the most advanced LLMs like GPT-4 lag behind human performance by over 10% points, indicating that LLMs have not achieved a human-level theory of mind yet. Our aim with ToMBench is to enable an efficient and effective evaluation of LLMs' ToM capabilities, thereby facilitating the development of LLMs with inherent social intelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15052 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ToMBench: Benchmarking Theory of Mind in Large Language Models Chen, Zhuang Wu, Jincenzi Zhou, Jinfeng Wen, Bosi Bi, Guanqun Jiang, Gongyao Cao, Yaru Hu, Mengting Lai, Yunghwei Xiong, Zexuan Huang, Minlie Computation and Language Artificial Intelligence Theory of Mind (ToM) is the cognitive capability to perceive and ascribe mental states to oneself and others. Recent research has sparked a debate over whether large language models (LLMs) exhibit a form of ToM. However, existing ToM evaluations are hindered by challenges such as constrained scope, subjective judgment, and unintended contamination, yielding inadequate assessments. To address this gap, we introduce ToMBench with three key characteristics: a systematic evaluation framework encompassing 8 tasks and 31 abilities in social cognition, a multiple-choice question format to support automated and unbiased evaluation, and a build-from-scratch bilingual inventory to strictly avoid data leakage. Based on ToMBench, we conduct extensive experiments to evaluate the ToM performance of 10 popular LLMs across tasks and abilities. We find that even the most advanced LLMs like GPT-4 lag behind human performance by over 10% points, indicating that LLMs have not achieved a human-level theory of mind yet. Our aim with ToMBench is to enable an efficient and effective evaluation of LLMs' ToM capabilities, thereby facilitating the development of LLMs with inherent social intelligence. |
| title | ToMBench: Benchmarking Theory of Mind in Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.15052 |