Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2502.14119 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915161643679744 |
|---|---|
| author | Zhu, Xiaomeng Zhou, Zhenghao Charlow, Simon Frank, Robert |
| author_facet | Zhu, Xiaomeng Zhou, Zhenghao Charlow, Simon Frank, Robert |
| contents | We present a hierarchy of natural language understanding abilities and argue for the importance of moving beyond assessments of understanding at the lexical and sentence levels to the discourse level. We propose the task of anaphora accessibility as a diagnostic for assessing discourse understanding, and to this end, present an evaluation dataset inspired by theoretical research in dynamic semantics. We evaluate human and LLM performance on our dataset and find that LLMs and humans align on some tasks and diverge on others. Such divergence can be explained by LLMs' reliance on specific lexical items during language comprehension, in contrast to human sensitivity to structural abstractions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14119 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Meaning Beyond Truth Conditions: Evaluating Discourse Level Understanding via Anaphora Accessibility Zhu, Xiaomeng Zhou, Zhenghao Charlow, Simon Frank, Robert Computation and Language We present a hierarchy of natural language understanding abilities and argue for the importance of moving beyond assessments of understanding at the lexical and sentence levels to the discourse level. We propose the task of anaphora accessibility as a diagnostic for assessing discourse understanding, and to this end, present an evaluation dataset inspired by theoretical research in dynamic semantics. We evaluate human and LLM performance on our dataset and find that LLMs and humans align on some tasks and diverge on others. Such divergence can be explained by LLMs' reliance on specific lexical items during language comprehension, in contrast to human sensitivity to structural abstractions. |
| title | Meaning Beyond Truth Conditions: Evaluating Discourse Level Understanding via Anaphora Accessibility |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.14119 |