Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.12057 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916456662302720 |
|---|---|
| author | Jacobs, Cassandra L. Grobol, Loïc Tsang, Alvin |
| author_facet | Jacobs, Cassandra L. Grobol, Loïc Tsang, Alvin |
| contents | In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that while large models trained for longer are typically better estimators of human productions, but they reliably under-estimate the probabilities of human responses, over-rank rare responses, under-rank top responses, and produce highly distinct semantic spaces. Altogether, this work demonstrates in a tractable, interpretable domain that LM generations can not be used as replacements of or models of the cloze task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12057 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned Jacobs, Cassandra L. Grobol, Loïc Tsang, Alvin Computation and Language Artificial Intelligence I.2.7 In this work we compare the generative behavior at the next token prediction level in several language models by comparing them to human productions in the cloze task. We find that while large models trained for longer are typically better estimators of human productions, but they reliably under-estimate the probabilities of human responses, over-rank rare responses, under-rank top responses, and produce highly distinct semantic spaces. Altogether, this work demonstrates in a tractable, interpretable domain that LM generations can not be used as replacements of or models of the cloze task. |
| title | Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned |
| topic | Computation and Language Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2410.12057 |