Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916916931592192 |
|---|---|
| author | Eckman, Stephanie Ma, Bolei Kern, Christoph Chew, Rob Plank, Barbara Kreuter, Frauke |
| author_facet | Eckman, Stephanie Ma, Bolei Kern, Christoph Chew, Rob Plank, Barbara Kreuter, Frauke |
| contents | Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR: Population-Aligned Instance Replication, a post-processing method that adjusts training data to better reflect target population characteristics without collecting additional annotations. Using simulation studies on offensive language and hate speech detection with varying annotator compositions, we show that non-representative pools degrade model calibration while leaving accuracy largely unchanged. PAIR corrects these calibration problems by replicating annotations from underrepresented annotator groups to match population proportions. We conclude with recommendations for improving the representativity of training data and model performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06826 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication Eckman, Stephanie Ma, Bolei Kern, Christoph Chew, Rob Plank, Barbara Kreuter, Frauke Methodology Computation and Language Models trained on crowdsourced annotations may not reflect population views, if those who work as annotators do not represent the broader population. In this paper, we propose PAIR: Population-Aligned Instance Replication, a post-processing method that adjusts training data to better reflect target population characteristics without collecting additional annotations. Using simulation studies on offensive language and hate speech detection with varying annotator compositions, we show that non-representative pools degrade model calibration while leaving accuracy largely unchanged. PAIR corrects these calibration problems by replicating annotations from underrepresented annotator groups to match population proportions. We conclude with recommendations for improving the representativity of training data and model performance. |
| title | Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication |
| topic | Methodology Computation and Language |
| url | https://arxiv.org/abs/2501.06826 |