Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.12679 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910493430513664 |
|---|---|
| author | Aich, Ankit Liu, Tingting Giorgi, Salvatore Isman, Kelsey Ungar, Lyle Curtis, Brenda |
| author_facet | Aich, Ankit Liu, Tingting Giorgi, Salvatore Isman, Kelsey Ungar, Lyle Curtis, Brenda |
| contents | Large Language Models (LLMs) are increasingly being used in educational and learning applications. Research has demonstrated that controlling for style, to fit the needs of the learner, fosters increased understanding, promotes inclusion, and helps with knowledge distillation. To understand the capabilities and limitations of contemporary LLMs in style control, we evaluated five state-of-the-art models: GPT-3.5, GPT-4, GPT-4o, Llama-3, and Mistral-instruct- 7B across two style control tasks. We observed significant inconsistencies in the first task, with model performances averaging between 5th and 8th grade reading levels for tasks intended for first-graders, and standard deviations up to 27.6. For our second task, we observed a statistically significant improvement in performance from 0.02 to 0.26. However, we find that even without stereotypes in reference texts, LLMs often generated culturally insensitive content during their tasks. We provide a thorough analysis and discussion of the results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12679 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Vernacular? I Barely Know Her: Challenges with Style Control and Stereotyping Aich, Ankit Liu, Tingting Giorgi, Salvatore Isman, Kelsey Ungar, Lyle Curtis, Brenda Computation and Language Large Language Models (LLMs) are increasingly being used in educational and learning applications. Research has demonstrated that controlling for style, to fit the needs of the learner, fosters increased understanding, promotes inclusion, and helps with knowledge distillation. To understand the capabilities and limitations of contemporary LLMs in style control, we evaluated five state-of-the-art models: GPT-3.5, GPT-4, GPT-4o, Llama-3, and Mistral-instruct- 7B across two style control tasks. We observed significant inconsistencies in the first task, with model performances averaging between 5th and 8th grade reading levels for tasks intended for first-graders, and standard deviations up to 27.6. For our second task, we observed a statistically significant improvement in performance from 0.02 to 0.26. However, we find that even without stereotypes in reference texts, LLMs often generated culturally insensitive content during their tasks. We provide a thorough analysis and discussion of the results. |
| title | Vernacular? I Barely Know Her: Challenges with Style Control and Stereotyping |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.12679 |