Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.00788 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913969898258432 |
|---|---|
| author | Tang, Xushuo Ding, Yi Yang, Zhengyi Chen, Yin Gu, Yongrui Yang, Wenke Ju, Mingchen Cao, Xin Liu, Yongfei Zhang, Wenjie |
| author_facet | Tang, Xushuo Ding, Yi Yang, Zhengyi Chen, Yin Gu, Yongrui Yang, Wenke Ju, Mingchen Cao, Xin Liu, Yongfei Zhang, Wenjie |
| contents | Large language models (LLMs) are increasingly deployed in sensitive contexts where fairness and inclusivity are critical. Pronoun usage, especially concerning gender-neutral and neopronouns, remains a key challenge for responsible AI. Prior work, such as the MISGENDERED benchmark, revealed significant limitations in earlier LLMs' handling of inclusive pronouns, but was constrained to outdated models and limited evaluations. In this study, we introduce MISGENDERED+, an extended and updated benchmark for evaluating LLMs' pronoun fidelity. We benchmark five representative LLMs, GPT-4o, Claude 4, DeepSeek-V3, Qwen Turbo, and Qwen2.5, across zero-shot, few-shot, and gender identity inference. Our results show notable improvements compared with previous studies, especially in binary and gender-neutral pronoun accuracy. However, accuracy on neopronouns and reverse inference tasks remains inconsistent, underscoring persistent gaps in identity-sensitive reasoning. We discuss implications, model-specific observations, and avenues for future inclusive AI research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_00788 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do They Understand Them? An Updated Evaluation on Nonbinary Pronoun Handling in Large Language Models Tang, Xushuo Ding, Yi Yang, Zhengyi Chen, Yin Gu, Yongrui Yang, Wenke Ju, Mingchen Cao, Xin Liu, Yongfei Zhang, Wenjie Computation and Language Artificial Intelligence Large language models (LLMs) are increasingly deployed in sensitive contexts where fairness and inclusivity are critical. Pronoun usage, especially concerning gender-neutral and neopronouns, remains a key challenge for responsible AI. Prior work, such as the MISGENDERED benchmark, revealed significant limitations in earlier LLMs' handling of inclusive pronouns, but was constrained to outdated models and limited evaluations. In this study, we introduce MISGENDERED+, an extended and updated benchmark for evaluating LLMs' pronoun fidelity. We benchmark five representative LLMs, GPT-4o, Claude 4, DeepSeek-V3, Qwen Turbo, and Qwen2.5, across zero-shot, few-shot, and gender identity inference. Our results show notable improvements compared with previous studies, especially in binary and gender-neutral pronoun accuracy. However, accuracy on neopronouns and reverse inference tasks remains inconsistent, underscoring persistent gaps in identity-sensitive reasoning. We discuss implications, model-specific observations, and avenues for future inclusive AI research. |
| title | Do They Understand Them? An Updated Evaluation on Nonbinary Pronoun Handling in Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.00788 |