Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915193707036672 |
|---|---|
| author | Yu, Liu Guo, Ludie Kuang, Ping Zhou, Fan |
| author_facet | Yu, Liu Guo, Ludie Kuang, Ping Zhou, Fan |
| contents | Pre-trained language models (PLMs) are trained on data that inherently contains gender biases, leading to undesirable impacts. Traditional debiasing methods often rely on external corpora, which may lack quality, diversity, or demographic balance, affecting the effectiveness of debiasing. With the rise of large language models and their extensive knowledge, we propose enhancing fairness (Fair-Gender) in PLMs by absorbing coherent, attribute-balanced, and semantically rich sentences. However, these sentences cannot be directly used for debiasing due to alignment issues and the risk of negative transfer. We address this by applying causal analysis to estimate causal effects, filtering out unaligned sentences, and identifying aligned ones for incorporation into PLMs, thereby ensuring positive transfer. Experiments show that our approach significantly reduces gender biases in PLMs while preserving their language expressiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06795 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences Yu, Liu Guo, Ludie Kuang, Ping Zhou, Fan Computation and Language Artificial Intelligence Pre-trained language models (PLMs) are trained on data that inherently contains gender biases, leading to undesirable impacts. Traditional debiasing methods often rely on external corpora, which may lack quality, diversity, or demographic balance, affecting the effectiveness of debiasing. With the rise of large language models and their extensive knowledge, we propose enhancing fairness (Fair-Gender) in PLMs by absorbing coherent, attribute-balanced, and semantically rich sentences. However, these sentences cannot be directly used for debiasing due to alignment issues and the risk of negative transfer. We address this by applying causal analysis to estimate causal effects, filtering out unaligned sentences, and identifying aligned ones for incorporation into PLMs, thereby ensuring positive transfer. Experiments show that our approach significantly reduces gender biases in PLMs while preserving their language expressiveness. |
| title | Bridging the Fairness Gap: Enhancing Pre-trained Models with LLM-Generated Sentences |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.06795 |