Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913383496810496 |
|---|---|
| author | Soubki, Adil Murzaku, John Jordehi, Arash Yousefi Zeng, Peter Markowska, Magdalena Mirroshandel, Seyed Abolghasem Rambow, Owen |
| author_facet | Soubki, Adil Murzaku, John Jordehi, Arash Yousefi Zeng, Peter Markowska, Magdalena Mirroshandel, Seyed Abolghasem Rambow, Owen |
| contents | Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received a great deal of attention. However, many existing benchmarks rely on synthetic data, which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_02451 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground Soubki, Adil Murzaku, John Jordehi, Arash Yousefi Zeng, Peter Markowska, Magdalena Mirroshandel, Seyed Abolghasem Rambow, Owen Computation and Language Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received a great deal of attention. However, many existing benchmarks rely on synthetic data, which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM. |
| title | Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.02451 |