Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915528566636544 |
|---|---|
| author | Pan, Eileen Choi, Anna Seo Gyeong ter Hoeve, Maartje Seto, Skyler Koenecke, Allison |
| author_facet | Pan, Eileen Choi, Anna Seo Gyeong ter Hoeve, Maartje Seto, Skyler Koenecke, Allison |
| contents | Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00962 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks Pan, Eileen Choi, Anna Seo Gyeong ter Hoeve, Maartje Seto, Skyler Koenecke, Allison Computation and Language Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures. |
| title | Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.00962 |