DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915172482809856 |
|---|---|
| author | Li, Zihao Tang, Ruixiang Cheng, Lu Wang, Shuaiqiang Yin, Dawei Du, Mengnan |
| author_facet | Li, Zihao Tang, Ruixiang Cheng, Lu Wang, Shuaiqiang Yin, Dawei Du, Mengnan |
| contents | Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks. However, recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language, especially for natural language understanding (NLU) tasks. Consequently, the models struggle to generalize to out-of-domain data. In this work, we propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior. Our method measures the divergence between the output distributions for original examples and examples where shortcut tokens have been masked. This process prevents the model's predictions from being overly influenced by shortcut features or biases. We evaluate our model on three NLU tasks and find that it improves out-of-domain performance with little loss of in-domain accuracy. Our results demonstrate that reducing the reliance on shortcuts and superficial features can enhance the generalization ability of large pre-trained language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18353 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models Li, Zihao Tang, Ruixiang Cheng, Lu Wang, Shuaiqiang Yin, Dawei Du, Mengnan Computation and Language Machine Learning Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks. However, recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language, especially for natural language understanding (NLU) tasks. Consequently, the models struggle to generalize to out-of-domain data. In this work, we propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior. Our method measures the divergence between the output distributions for original examples and examples where shortcut tokens have been masked. This process prevents the model's predictions from being overly influenced by shortcut features or biases. We evaluate our model on three NLU tasks and find that it improves out-of-domain performance with little loss of in-domain accuracy. Our results demonstrate that reducing the reliance on shortcuts and superficial features can enhance the generalization ability of large pre-trained language models. |
| title | DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.18353 |