Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915008028344320 |
|---|---|
| author | Jung, Chani Kim, Dongkwan Jin, Jiho Kim, Jiseon Seonwoo, Yeon Choi, Yejin Oh, Alice Kim, Hyunwoo |
| author_facet | Jung, Chani Kim, Dongkwan Jin, Jiho Kim, Jiseon Seonwoo, Yeon Choi, Yejin Oh, Alice Kim, Hyunwoo |
| contents | While humans naturally develop theory of mind (ToM), the capability to understand other people's mental states and beliefs, state-of-the-art large language models (LLMs) underperform on simple ToM benchmarks. We posit that we can extend our understanding of LLMs' ToM abilities by evaluating key human ToM precursors$-$perception inference and perception-to-belief inference$-$in LLMs. We introduce two datasets, Percept-ToMi and Percept-FANToM, to evaluate these precursory inferences for ToM in LLMs by annotating characters' perceptions on ToMi and FANToM, respectively. Our evaluation of eight state-of-the-art LLMs reveals that the models generally perform well in perception inference while exhibiting limited capability in perception-to-belief inference (e.g., lack of inhibitory control). Based on these results, we present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference. Experimental results demonstrate that PercepToM significantly enhances LLM's performance, especially in false belief scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06004 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models Jung, Chani Kim, Dongkwan Jin, Jiho Kim, Jiseon Seonwoo, Yeon Choi, Yejin Oh, Alice Kim, Hyunwoo Computation and Language While humans naturally develop theory of mind (ToM), the capability to understand other people's mental states and beliefs, state-of-the-art large language models (LLMs) underperform on simple ToM benchmarks. We posit that we can extend our understanding of LLMs' ToM abilities by evaluating key human ToM precursors$-$perception inference and perception-to-belief inference$-$in LLMs. We introduce two datasets, Percept-ToMi and Percept-FANToM, to evaluate these precursory inferences for ToM in LLMs by annotating characters' perceptions on ToMi and FANToM, respectively. Our evaluation of eight state-of-the-art LLMs reveals that the models generally perform well in perception inference while exhibiting limited capability in perception-to-belief inference (e.g., lack of inhibitory control). Based on these results, we present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference. Experimental results demonstrate that PercepToM significantly enhances LLM's performance, especially in false belief scenarios. |
| title | Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.06004 |