Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908681497477120 |
|---|---|
| author | Chen, Nuo Fang, Hanpei Liu, Jiqun Wei, Wilson Sakai, Tetsuya Wu, Xiao-Ming |
| author_facet | Chen, Nuo Fang, Hanpei Liu, Jiqun Wei, Wilson Sakai, Tetsuya Wu, Xiao-Ming |
| contents | Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence later ones. Although psychological theories link personality traits to such biases, it is unclear whether simulated personalities in LLMs exhibit similar effects. We investigate how Big Five personality profiles in LLMs influence priming in relevance labeling, using multiple LLMs on TREC 2021 and 2022 Deep Learning Track datasets. Our results show that certain profiles, such as High Openness and Low Neuroticism, consistently reduce priming susceptibility. Additionally, the most effective personality in mitigating priming may vary across models and task types. Based on these findings, we propose personality prompting as a method to mitigate threshold priming, connecting psychological evidence with LLM-based evaluation practices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00390 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing Chen, Nuo Fang, Hanpei Liu, Jiqun Wei, Wilson Sakai, Tetsuya Wu, Xiao-Ming Computation and Language Information Retrieval Recent research has explored LLMs as scalable tools for relevance labeling, but studies indicate they are susceptible to priming effects, where prior relevance judgments influence later ones. Although psychological theories link personality traits to such biases, it is unclear whether simulated personalities in LLMs exhibit similar effects. We investigate how Big Five personality profiles in LLMs influence priming in relevance labeling, using multiple LLMs on TREC 2021 and 2022 Deep Learning Track datasets. Our results show that certain profiles, such as High Openness and Low Neuroticism, consistently reduce priming susceptibility. Additionally, the most effective personality in mitigating priming may vary across models and task types. Based on these findings, we propose personality prompting as a method to mitigate threshold priming, connecting psychological evidence with LLM-based evaluation practices. |
| title | Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing |
| topic | Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2512.00390 |