Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915335316176896 |
|---|---|
| author | Yuan, Shuzhou Nie, Ercong Tawfelis, Mario Schmid, Helmut Schütze, Hinrich Färber, Michael |
| author_facet | Yuan, Shuzhou Nie, Ercong Tawfelis, Mario Schmid, Helmut Schütze, Hinrich Färber, Michael |
| contents | Hate speech detection is a socially sensitive and inherently subjective task, with judgments often varying based on personal traits. While prior work has examined how socio-demographic factors influence annotation, the impact of personality traits on Large Language Models (LLMs) remains largely unexplored. In this paper, we present the first comprehensive study on the role of persona prompts in hate speech classification, focusing on MBTI-based traits. A human annotation survey confirms that MBTI dimensions significantly affect labeling behavior. Extending this to LLMs, we prompt four open-source models with MBTI personas and evaluate their outputs across three hate speech datasets. Our analysis uncovers substantial persona-driven variation, including inconsistencies with ground truth, inter-persona disagreement, and logit-level biases. These findings highlight the need to carefully define persona prompts in LLM-based annotation workflows, with implications for fairness and alignment with human values. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08593 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models Yuan, Shuzhou Nie, Ercong Tawfelis, Mario Schmid, Helmut Schütze, Hinrich Färber, Michael Computation and Language Hate speech detection is a socially sensitive and inherently subjective task, with judgments often varying based on personal traits. While prior work has examined how socio-demographic factors influence annotation, the impact of personality traits on Large Language Models (LLMs) remains largely unexplored. In this paper, we present the first comprehensive study on the role of persona prompts in hate speech classification, focusing on MBTI-based traits. A human annotation survey confirms that MBTI dimensions significantly affect labeling behavior. Extending this to LLMs, we prompt four open-source models with MBTI personas and evaluate their outputs across three hate speech datasets. Our analysis uncovers substantial persona-driven variation, including inconsistencies with ground truth, inter-persona disagreement, and logit-level biases. These findings highlight the need to carefully define persona prompts in LLM-based annotation workflows, with implications for fairness and alignment with human values. |
| title | Hateful Person or Hateful Model? Investigating the Role of Personas in Hate Speech Detection by Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.08593 |