Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10789 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912501797486592 |
|---|---|
| author | Deas, Nicholas Vente, Blake Ananthram, Amith Grieser, Jessica A. Patton, Desmond Kleiner, Shana Shepard, James McKeown, Kathleen |
| author_facet | Deas, Nicholas Vente, Blake Ananthram, Amith Grieser, Jessica A. Patton, Desmond Kleiner, Shana Shepard, James McKeown, Kathleen |
| contents | With a combination of quantitative experiments, human judgments, and qualitative analyses, we evaluate the quantity and quality of African American Language (AAL) representation in 12 predominantly English, open-source pretraining corpora. We specifically focus on the sources, variation, and naturalness of included AAL texts representing the AAL-speaking community. We find that AAL is underrepresented in all evaluated pretraining corpora compared to US demographics, constituting as few as 0.007% and at most 0.18% of documents. We also find that more than 25% of AAL texts in C4 may be perceived as inappropriate for LLMs to generate and to reinforce harmful stereotypes. Finally, we find that most automated filters are more likely to conserve White Mainstream English (WME) texts over AAL in pretraining corpora. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10789 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Data Caricatures: On the Representation of African American Language in Pretraining Corpora Deas, Nicholas Vente, Blake Ananthram, Amith Grieser, Jessica A. Patton, Desmond Kleiner, Shana Shepard, James McKeown, Kathleen Computation and Language With a combination of quantitative experiments, human judgments, and qualitative analyses, we evaluate the quantity and quality of African American Language (AAL) representation in 12 predominantly English, open-source pretraining corpora. We specifically focus on the sources, variation, and naturalness of included AAL texts representing the AAL-speaking community. We find that AAL is underrepresented in all evaluated pretraining corpora compared to US demographics, constituting as few as 0.007% and at most 0.18% of documents. We also find that more than 25% of AAL texts in C4 may be perceived as inappropriate for LLMs to generate and to reinforce harmful stereotypes. Finally, we find that most automated filters are more likely to conserve White Mainstream English (WME) texts over AAL in pretraining corpora. |
| title | Data Caricatures: On the Representation of African American Language in Pretraining Corpora |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.10789 |