A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913557863464960 |
|---|---|
| author | Lee, Eun-Kyoung Rosa Nair, Sathvik Feldman, Naomi |
| author_facet | Lee, Eun-Kyoung Rosa Nair, Sathvik Feldman, Naomi |
| contents | We present a systematic evaluation of large language models' sensitivity to argument roles, i.e., who did what to whom, by replicating psycholinguistic studies on human argument role processing. In three experiments, we find that language models are able to distinguish verbs that appear in plausible and implausible contexts, where plausibility is determined through the relation between the verb and its preceding arguments. However, none of the models capture the same selective patterns that human comprehenders exhibit during real-time verb prediction. This indicates that language models' capacity to detect verb plausibility does not arise from the same mechanism that underlies human real-time sentence processing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16139 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles Lee, Eun-Kyoung Rosa Nair, Sathvik Feldman, Naomi Computation and Language We present a systematic evaluation of large language models' sensitivity to argument roles, i.e., who did what to whom, by replicating psycholinguistic studies on human argument role processing. In three experiments, we find that language models are able to distinguish verbs that appear in plausible and implausible contexts, where plausibility is determined through the relation between the verb and its preceding arguments. However, none of the models capture the same selective patterns that human comprehenders exhibit during real-time verb prediction. This indicates that language models' capacity to detect verb plausibility does not arise from the same mechanism that underlies human real-time sentence processing. |
| title | A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.16139 |