Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909230474199040 |
|---|---|
| author | Chen, Shan Gallifant, Jack Gao, Mingye Moreira, Pedro Munch, Nikolaj Muthukkumar, Ajay Rajan, Arvind Kolluri, Jaya Fiske, Amelia Hastings, Janna Aerts, Hugo Anthony, Brian Celi, Leo Anthony La Cava, William G. Bitterman, Danielle S. |
| author_facet | Chen, Shan Gallifant, Jack Gao, Mingye Moreira, Pedro Munch, Nikolaj Muthukkumar, Ajay Rajan, Arvind Kolluri, Jaya Fiske, Amelia Hastings, Janna Aerts, Hugo Anthony, Brian Celi, Leo Anthony La Cava, William G. Bitterman, Danielle S. |
| contents | Large language models (LLMs) are increasingly essential in processing natural languages, yet their application is frequently compromised by biases and inaccuracies originating in their training data. In this study, we introduce Cross-Care, the first benchmark framework dedicated to assessing biases and real world knowledge in LLMs, specifically focusing on the representation of disease prevalence across diverse demographic groups. We systematically evaluate how demographic biases embedded in pre-training corpora like $ThePile$ influence the outputs of LLMs. We expose and quantify discrepancies by juxtaposing these biases against actual disease prevalences in various U.S. demographic groups. Our results highlight substantial misalignment between LLM representation of disease prevalence and real disease prevalence rates across demographic subgroups, indicating a pronounced risk of bias propagation and a lack of real-world grounding for medical applications of LLMs. Furthermore, we observe that various alignment methods minimally resolve inconsistencies in the models' representation of disease prevalence across different languages. For further exploration and analysis, we make all data and a data visualization tool available at: www.crosscare.net. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_05506 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias Chen, Shan Gallifant, Jack Gao, Mingye Moreira, Pedro Munch, Nikolaj Muthukkumar, Ajay Rajan, Arvind Kolluri, Jaya Fiske, Amelia Hastings, Janna Aerts, Hugo Anthony, Brian Celi, Leo Anthony La Cava, William G. Bitterman, Danielle S. Computation and Language Large language models (LLMs) are increasingly essential in processing natural languages, yet their application is frequently compromised by biases and inaccuracies originating in their training data. In this study, we introduce Cross-Care, the first benchmark framework dedicated to assessing biases and real world knowledge in LLMs, specifically focusing on the representation of disease prevalence across diverse demographic groups. We systematically evaluate how demographic biases embedded in pre-training corpora like $ThePile$ influence the outputs of LLMs. We expose and quantify discrepancies by juxtaposing these biases against actual disease prevalences in various U.S. demographic groups. Our results highlight substantial misalignment between LLM representation of disease prevalence and real disease prevalence rates across demographic subgroups, indicating a pronounced risk of bias propagation and a lack of real-world grounding for medical applications of LLMs. Furthermore, we observe that various alignment methods minimally resolve inconsistencies in the models' representation of disease prevalence across different languages. For further exploration and analysis, we make all data and a data visualization tool available at: www.crosscare.net. |
| title | Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.05506 |