Fairness Evaluation of Large Language Models in Academic Library Reference Services
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915630419017728 |
|---|---|
| author | Wang, Haining Clark, Jason Yan, Yueru Bradley, Star Chen, Ruiyang Zhang, Yiqiong Fu, Hengyi Tian, Zuoyu |
| author_facet | Wang, Haining Clark, Jason Yan, Yueru Bradley, Star Chen, Ruiyang Zhang, Yiqiong Fu, Hengyi Tian, Zuoyu |
| contents | As libraries explore large language models (LLMs) for use in virtual reference services, a key question arises: Can LLMs serve all users equitably, regardless of demographics or social status? While they offer great potential for scalable support, LLMs may also reproduce societal biases embedded in their training data, risking the integrity of libraries' commitment to equitable service. To address this concern, we evaluate whether LLMs differentiate responses across user identities by prompting six state-of-the-art LLMs to assist patrons differing in sex, race/ethnicity, and institutional role. We find no evidence of differentiation by race or ethnicity, and only minor evidence of stereotypical bias against women in one model. LLMs demonstrate nuanced accommodation of institutional roles through the use of linguistic choices related to formality, politeness, and domain-specific vocabularies, reflecting professional norms rather than discriminatory treatment. These findings suggest that current LLMs show a promising degree of readiness to support equitable and contextually appropriate communication in academic library reference services. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04224 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fairness Evaluation of Large Language Models in Academic Library Reference Services Wang, Haining Clark, Jason Yan, Yueru Bradley, Star Chen, Ruiyang Zhang, Yiqiong Fu, Hengyi Tian, Zuoyu Computation and Language Artificial Intelligence Digital Libraries As libraries explore large language models (LLMs) for use in virtual reference services, a key question arises: Can LLMs serve all users equitably, regardless of demographics or social status? While they offer great potential for scalable support, LLMs may also reproduce societal biases embedded in their training data, risking the integrity of libraries' commitment to equitable service. To address this concern, we evaluate whether LLMs differentiate responses across user identities by prompting six state-of-the-art LLMs to assist patrons differing in sex, race/ethnicity, and institutional role. We find no evidence of differentiation by race or ethnicity, and only minor evidence of stereotypical bias against women in one model. LLMs demonstrate nuanced accommodation of institutional roles through the use of linguistic choices related to formality, politeness, and domain-specific vocabularies, reflecting professional norms rather than discriminatory treatment. These findings suggest that current LLMs show a promising degree of readiness to support equitable and contextually appropriate communication in academic library reference services. |
| title | Fairness Evaluation of Large Language Models in Academic Library Reference Services |
| topic | Computation and Language Artificial Intelligence Digital Libraries |
| url | https://arxiv.org/abs/2507.04224 |