Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914458210664448 |
|---|---|
| author | Barkhordar, Ehsan Safa, Abdulfattah Blaschke, Verena Lombart, Erika de Marneffe, Marie-Catherine Şahin, Gözde Gül |
| author_facet | Barkhordar, Ehsan Safa, Abdulfattah Blaschke, Verena Lombart, Erika de Marneffe, Marie-Catherine Şahin, Gözde Gül |
| contents | Peer review plays a central role in the NLP publication process, but is susceptible to various biases. Here, we study language-of-study (LoS) bias: the tendency for reviewers to evaluate a paper differently based on the language(s) it studies, rather than its scientific merit. Despite being explicitly flagged in reviewing guidelines, such biases are poorly understood. Prior work treats such comments as part of broader categories of weak or unconstructive reviews without defining them as a distinct form of bias. We present the first systematic characterization of LoS bias, distinguishing negative and positive forms, and introduce the human-annotated dataset LOBSTER (Language-Of-study Bias in ScienTific pEer Review) and a method achieving 87.37 macro F1 for detection. We analyze 15,645 reviews to estimate how negative and positive biases differ with respect to the LoS, and find that non-English papers face substantially higher bias rates than English-only ones, with negative bias consistently outweighing positive bias. Finally, we identify four subcategories of negative bias, and find that demanding unjustified cross-lingual generalization is the most dominant form. We publicly release all resources to support work on fairer reviewing practices in NLP and beyond. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07119 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews Barkhordar, Ehsan Safa, Abdulfattah Blaschke, Verena Lombart, Erika de Marneffe, Marie-Catherine Şahin, Gözde Gül Computation and Language I.2.7 Peer review plays a central role in the NLP publication process, but is susceptible to various biases. Here, we study language-of-study (LoS) bias: the tendency for reviewers to evaluate a paper differently based on the language(s) it studies, rather than its scientific merit. Despite being explicitly flagged in reviewing guidelines, such biases are poorly understood. Prior work treats such comments as part of broader categories of weak or unconstructive reviews without defining them as a distinct form of bias. We present the first systematic characterization of LoS bias, distinguishing negative and positive forms, and introduce the human-annotated dataset LOBSTER (Language-Of-study Bias in ScienTific pEer Review) and a method achieving 87.37 macro F1 for detection. We analyze 15,645 reviews to estimate how negative and positive biases differ with respect to the LoS, and find that non-English papers face substantially higher bias rates than English-only ones, with negative bias consistently outweighing positive bias. Finally, we identify four subcategories of negative bias, and find that demanding unjustified cross-lingual generalization is the most dominant form. We publicly release all resources to support work on fairer reviewing practices in NLP and beyond. |
| title | Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2604.07119 |