LLMs achieve adult human performance on higher-order theory of mind tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913371654193152 |
|---|---|
| author | Street, Winnie Siy, John Oliver Keeling, Geoff Baranes, Adrien Barnett, Benjamin McKibben, Michael Kanyere, Tatenda Lentz, Alison Arcas, Blaise Aguera y Dunbar, Robin I. M. |
| author_facet | Street, Winnie Siy, John Oliver Keeling, Geoff Baranes, Adrien Barnett, Benjamin McKibben, Michael Kanyere, Tatenda Lentz, Alison Arcas, Blaise Aguera y Dunbar, Robin I. M. |
| contents | This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (e.g. I think that you believe that she knows). This paper builds on prior work by introducing a handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to compare the performance of five LLMs to a newly gathered adult human benchmark. We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level performance on ToM tasks overall, and that GPT-4 exceeds adult performance on 6th order inferences. Our results suggest that there is an interplay between model size and finetuning for the realisation of ToM abilities, and that the best-performing LLMs have developed a generalised capacity for ToM. Given the role that higher-order ToM plays in a wide range of cooperative and competitive human behaviours, these findings have significant implications for user-facing LLM applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_18870 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLMs achieve adult human performance on higher-order theory of mind tasks Street, Winnie Siy, John Oliver Keeling, Geoff Baranes, Adrien Barnett, Benjamin McKibben, Michael Kanyere, Tatenda Lentz, Alison Arcas, Blaise Aguera y Dunbar, Robin I. M. Artificial Intelligence Computation and Language Human-Computer Interaction I.2.7; H.1.2 This paper examines the extent to which large language models (LLMs) have developed higher-order theory of mind (ToM); the human ability to reason about multiple mental and emotional states in a recursive manner (e.g. I think that you believe that she knows). This paper builds on prior work by introducing a handwritten test suite -- Multi-Order Theory of Mind Q&A -- and using it to compare the performance of five LLMs to a newly gathered adult human benchmark. We find that GPT-4 and Flan-PaLM reach adult-level and near adult-level performance on ToM tasks overall, and that GPT-4 exceeds adult performance on 6th order inferences. Our results suggest that there is an interplay between model size and finetuning for the realisation of ToM abilities, and that the best-performing LLMs have developed a generalised capacity for ToM. Given the role that higher-order ToM plays in a wide range of cooperative and competitive human behaviours, these findings have significant implications for user-facing LLM applications. |
| title | LLMs achieve adult human performance on higher-order theory of mind tasks |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction I.2.7; H.1.2 |
| url | https://arxiv.org/abs/2405.18870 |