General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916105275047936 |
|---|---|
| author | Shi, Bingkang Zhang, Xiaodan Kong, Dehan Wu, Yulei Liu, Zongzhen Lyu, Honglei Huang, Longtao |
| author_facet | Shi, Bingkang Zhang, Xiaodan Kong, Dehan Wu, Yulei Liu, Zongzhen Lyu, Honglei Huang, Longtao |
| contents | The social biases and unwelcome stereotypes revealed by pretrained language models are becoming obstacles to their application. Compared to numerous debiasing methods targeting word level, there has been relatively less attention on biases present at phrase level, limiting the performance of debiasing in discipline domains. In this paper, we propose an automatic multi-token debiasing pipeline called \textbf{General Phrase Debiaser}, which is capable of mitigating phrase-level biases in masked language models. Specifically, our method consists of a \textit{phrase filter stage} that generates stereotypical phrases from Wikipedia pages as well as a \textit{model debias stage} that can debias models at the multi-token level to tackle bias challenges on phrases. The latter searches for prompts that trigger model's bias, and then uses them for debiasing. State-of-the-art results on standard datasets and metrics show that our approach can significantly reduce gender biases on both career and multiple disciplines, across models with varying parameter sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_13892 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level Shi, Bingkang Zhang, Xiaodan Kong, Dehan Wu, Yulei Liu, Zongzhen Lyu, Honglei Huang, Longtao Computation and Language Artificial Intelligence The social biases and unwelcome stereotypes revealed by pretrained language models are becoming obstacles to their application. Compared to numerous debiasing methods targeting word level, there has been relatively less attention on biases present at phrase level, limiting the performance of debiasing in discipline domains. In this paper, we propose an automatic multi-token debiasing pipeline called \textbf{General Phrase Debiaser}, which is capable of mitigating phrase-level biases in masked language models. Specifically, our method consists of a \textit{phrase filter stage} that generates stereotypical phrases from Wikipedia pages as well as a \textit{model debias stage} that can debias models at the multi-token level to tackle bias challenges on phrases. The latter searches for prompts that trigger model's bias, and then uses them for debiasing. State-of-the-art results on standard datasets and metrics show that our approach can significantly reduce gender biases on both career and multiple disciplines, across models with varying parameter sizes. |
| title | General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2311.13892 |