When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913689597116416 |
|---|---|
| author | Amouyal, Samuel Joseph Meltzer-Asscher, Aya Berant, Jonathan |
| author_facet | Amouyal, Samuel Joseph Meltzer-Asscher, Aya Berant, Jonathan |
| contents | Modern Large Language Models (LLMs) have shown human-like abilities in many language tasks, sparking interest in comparing LLMs' and humans' language processing. In this paper, we conduct a detailed comparison of the two on a sentence comprehension task using garden-path constructions, which are notoriously challenging for humans. Based on psycholinguistic research, we formulate hypotheses on why garden-path sentences are hard, and test these hypotheses on human participants and a large suite of LLMs using comprehension questions. Our findings reveal that both LLMs and humans struggle with specific syntactic complexities, with some models showing high correlation with human comprehension. To complement our findings, we test LLM comprehension of garden-path constructions with paraphrasing and text-to-image generation tasks, and find that the results mirror the sentence comprehension question results, further validating our findings on LLM understanding of these constructions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_09307 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models Amouyal, Samuel Joseph Meltzer-Asscher, Aya Berant, Jonathan Computation and Language Artificial Intelligence Modern Large Language Models (LLMs) have shown human-like abilities in many language tasks, sparking interest in comparing LLMs' and humans' language processing. In this paper, we conduct a detailed comparison of the two on a sentence comprehension task using garden-path constructions, which are notoriously challenging for humans. Based on psycholinguistic research, we formulate hypotheses on why garden-path sentences are hard, and test these hypotheses on human participants and a large suite of LLMs using comprehension questions. Our findings reveal that both LLMs and humans struggle with specific syntactic complexities, with some models showing high correlation with human comprehension. To complement our findings, we test LLM comprehension of garden-path constructions with paraphrasing and text-to-image generation tasks, and find that the results mirror the sentence comprehension question results, further validating our findings on LLM understanding of these constructions. |
| title | When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.09307 |