Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917986077507584 |
|---|---|
| author | Wang, Qiaosi Zhou, Xuhui Sap, Maarten Forlizzi, Jodi Shen, Hong |
| author_facet | Wang, Qiaosi Zhou, Xuhui Sap, Maarten Forlizzi, Jodi Shen, Hong |
| contents | The last couple of years have witnessed emerging research that appropriates Theory-of-Mind (ToM) tasks designed for humans to benchmark LLM's ToM capabilities as an indication of LLM's social intelligence. However, this approach has a number of limitations. Drawing on existing psychology and AI literature, we summarize the theoretical, methodological, and evaluation limitations by pointing out that certain issues are inherently present in the original ToM tasks used to evaluate human's ToM, which continues to persist and exacerbated when appropriated to benchmark LLM's ToM. Taking a human-computer interaction (HCI) perspective, these limitations prompt us to rethink the definition and criteria of ToM in ToM benchmarks in a more dynamic, interactional approach that accounts for user preferences, needs, and experiences with LLMs in such evaluations. We conclude by outlining potential opportunities and challenges towards this direction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_10839 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective Wang, Qiaosi Zhou, Xuhui Sap, Maarten Forlizzi, Jodi Shen, Hong Human-Computer Interaction Artificial Intelligence The last couple of years have witnessed emerging research that appropriates Theory-of-Mind (ToM) tasks designed for humans to benchmark LLM's ToM capabilities as an indication of LLM's social intelligence. However, this approach has a number of limitations. Drawing on existing psychology and AI literature, we summarize the theoretical, methodological, and evaluation limitations by pointing out that certain issues are inherently present in the original ToM tasks used to evaluate human's ToM, which continues to persist and exacerbated when appropriated to benchmark LLM's ToM. Taking a human-computer interaction (HCI) perspective, these limitations prompt us to rethink the definition and criteria of ToM in ToM benchmarks in a more dynamic, interactional approach that accounts for user preferences, needs, and experiences with LLMs in such evaluations. We conclude by outlining potential opportunities and challenges towards this direction. |
| title | Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2504.10839 |