Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917538799026176 |
|---|---|
| author | Sánchez-Guzmán, Annabella Eberhard, Lukas Helic, Denis Espín-Noboa, Lisette |
| author_facet | Sánchez-Guzmán, Annabella Eberhard, Lukas Helic, Denis Espín-Noboa, Lisette |
| contents | Large language models (LLMs) are increasingly used as scholar recommenders, shaping who is seen as an expert in academia. Existing audits remain English-centric, single discipline, and persona-agnostic, leaving the source of output variability poorly understood. To this end, we propose a benchmark that disentangles the effects of model choice and prompt design on recommendations. We audit 43 LLMs by varying persona prompts (language, location, role-and-task) and context (field, seniority, k). Recommended scholars are compared against Semantic Scholar over six scientific disciplines to measure technical quality (factuality, coverage) and social representativeness (diversity, parity). Basic technical quality is driven by model choice, factuality and parity by context, and diversity by location. South Africa prompts yield less factual lists, while Japan prompts yield highly factual but homogeneous lists skewed toward highly productive scholars. Prompt design is thus a non-trivial axis of LLM-based scholar discovery and should be systematically audited alongside model choice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_28187 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendation Sánchez-Guzmán, Annabella Eberhard, Lukas Helic, Denis Espín-Noboa, Lisette Information Retrieval Artificial Intelligence Computers and Society Social and Information Networks H.3.3; I.2.7 Large language models (LLMs) are increasingly used as scholar recommenders, shaping who is seen as an expert in academia. Existing audits remain English-centric, single discipline, and persona-agnostic, leaving the source of output variability poorly understood. To this end, we propose a benchmark that disentangles the effects of model choice and prompt design on recommendations. We audit 43 LLMs by varying persona prompts (language, location, role-and-task) and context (field, seniority, k). Recommended scholars are compared against Semantic Scholar over six scientific disciplines to measure technical quality (factuality, coverage) and social representativeness (diversity, parity). Basic technical quality is driven by model choice, factuality and parity by context, and diversity by location. South Africa prompts yield less factual lists, while Japan prompts yield highly factual but homogeneous lists skewed toward highly productive scholars. Prompt design is thus a non-trivial axis of LLM-based scholar discovery and should be systematically audited alongside model choice. |
| title | Whose Name Comes Up? III: Persona Prompting Effects in LLM-Based Scholar Recommendation |
| topic | Information Retrieval Artificial Intelligence Computers and Society Social and Information Networks H.3.3; I.2.7 |
| url | https://arxiv.org/abs/2605.28187 |