Language models align with human judgments on key grammatical constructions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929478962249728 |
|---|---|
| author | Hu, Jennifer Mahowald, Kyle Lupyan, Gary Ivanova, Anna Levy, Roger |
| author_facet | Hu, Jennifer Mahowald, Kyle Lupyan, Gary Ivanova, Anna Levy, Roger |
| contents | Do large language models (LLMs) make human-like linguistic generalizations? Dentella et al. (2023) ("DGL") prompt several LLMs ("Is the following sentence grammatically correct in English?") to elicit grammaticality judgments of 80 English sentences, concluding that LLMs demonstrate a "yes-response bias" and a "failure to distinguish grammatical from ungrammatical sentences". We re-evaluate LLM performance using well-established practices and find that DGL's data in fact provide evidence for just how well LLMs capture human behaviors. Models not only achieve high accuracy overall, but also capture fine-grained variation in human linguistic judgments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01676 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language models align with human judgments on key grammatical constructions Hu, Jennifer Mahowald, Kyle Lupyan, Gary Ivanova, Anna Levy, Roger Computation and Language Artificial Intelligence Do large language models (LLMs) make human-like linguistic generalizations? Dentella et al. (2023) ("DGL") prompt several LLMs ("Is the following sentence grammatically correct in English?") to elicit grammaticality judgments of 80 English sentences, concluding that LLMs demonstrate a "yes-response bias" and a "failure to distinguish grammatical from ungrammatical sentences". We re-evaluate LLM performance using well-established practices and find that DGL's data in fact provide evidence for just how well LLMs capture human behaviors. Models not only achieve high accuracy overall, but also capture fine-grained variation in human linguistic judgments. |
| title | Language models align with human judgments on key grammatical constructions |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.01676 |