Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.13738 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915028960018432 |
|---|---|
| author | Bas, Tetiana |
| author_facet | Bas, Tetiana |
| contents | This study investigates gender bias in large language models (LLMs) by comparing their gender perception to that of human respondents, U.S. Bureau of Labor Statistics data, and a 50% no-bias benchmark. We created a new evaluation set using occupational data and role-specific sentences. Unlike common benchmarks included in LLM training data, our set is newly developed, preventing data leakage and test set contamination. Five LLMs were tested to predict the gender for each role using single-word answers. We used Kullback-Leibler (KL) divergence to compare model outputs with human perceptions, statistical data, and the 50% neutrality benchmark. All LLMs showed significant deviation from gender neutrality and aligned more with statistical data, still reflecting inherent biases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_13738 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics Bas, Tetiana Computation and Language Machine Learning This study investigates gender bias in large language models (LLMs) by comparing their gender perception to that of human respondents, U.S. Bureau of Labor Statistics data, and a 50% no-bias benchmark. We created a new evaluation set using occupational data and role-specific sentences. Unlike common benchmarks included in LLM training data, our set is newly developed, preventing data leakage and test set contamination. Five LLMs were tested to predict the gender for each role using single-word answers. We used Kullback-Leibler (KL) divergence to compare model outputs with human perceptions, statistical data, and the 50% neutrality benchmark. All LLMs showed significant deviation from gender neutrality and aligned more with statistical data, still reflecting inherent biases. |
| title | Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2411.13738 |