Do LLMs write like humans? Variation in grammatical and rhetorical styles
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916911567077376 |
|---|---|
| author | Reinhart, Alex Markey, Ben Laudenbach, Michael Pantusen, Kachatad Yurko, Ronald Weinberg, Gordon Brown, David West |
| author_facet | Reinhart, Alex Markey, Ben Laudenbach, Michael Pantusen, Kachatad Yurko, Ronald Weinberg, Gordon Brown, David West |
| contents | Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in surface features such as word choice and punctuation, and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs.
Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones, and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16107 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Do LLMs write like humans? Variation in grammatical and rhetorical styles Reinhart, Alex Markey, Ben Laudenbach, Michael Pantusen, Kachatad Yurko, Ronald Weinberg, Gordon Brown, David West Computation and Language Large language models (LLMs) are capable of writing grammatical text that follows instructions, answers questions, and solves problems. As they have advanced, it has become difficult to distinguish their output from human-written text. While past research has found some differences in surface features such as word choice and punctuation, and developed classifiers to detect LLM output, none has studied the rhetorical styles of LLMs. Using several variants of Llama 3 and GPT-4o, we construct two parallel corpora of human- and LLM-written texts from common prompts. Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans and between different LLMs. These differences persist when moving from smaller models to larger ones, and are larger for instruction-tuned models than base models. This observation of differences demonstrates that despite their advanced abilities, LLMs struggle to match human stylistic variation. Attention to more advanced linguistic features can hence detect patterns in their behavior not previously recognized. |
| title | Do LLMs write like humans? Variation in grammatical and rhetorical styles |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.16107 |