Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917508656660480 |
|---|---|
| author | Lu, Pinxian Ishfaq, Nimra Win, Emma Rose, Morgan Strickland, Sierra R Biernesser, Candice L Zelazny, Jamie De Choudhury, Munmun |
| author_facet | Lu, Pinxian Ishfaq, Nimra Win, Emma Rose, Morgan Strickland, Sierra R Biernesser, Candice L Zelazny, Jamie De Choudhury, Munmun |
| contents | Online harassment is a widespread social and public health concern, yet most computational approaches for detecting and addressing harassment focus on publicly visible social media content rather than private messaging environments. Private conversations present unique challenges because harmful interactions often unfold through context-dependent, multi-turn exchanges, while victims may lack timely support during moments of harassment. In this study, we investigate how large language models (LLMs) can support both the detection of and response to online harassment in private messaging. Using a dataset of 80,053 Instagram direct messages donated by 26 adolescents aged 12-18, including youth with suicide risk factors, we first construct a human-labeled dataset of online harassment in private conversations and develop a context-aware cascading LLM classification pipeline. The proposed pipeline outperforms baseline toxicity classifiers trained primarily on public social media data. We then develop a victim-centered response framework that produces context-sensitive and psychologically-grounded AI-generated responses to online harassment messages. Human evaluators perceived the AI-generated responses as significantly more helpful than the original participant responses (95% CI: 0.767--0.815, p < .001), particularly in terms of emotional support and de-escalation. Our findings highlight the potential of context-aware and victim-centered AI systems to provide just-in-time support during harassment in private messaging environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_14700 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging Lu, Pinxian Ishfaq, Nimra Win, Emma Rose, Morgan Strickland, Sierra R Biernesser, Candice L Zelazny, Jamie De Choudhury, Munmun Social and Information Networks Computation and Language Computers and Society 68T42 Online harassment is a widespread social and public health concern, yet most computational approaches for detecting and addressing harassment focus on publicly visible social media content rather than private messaging environments. Private conversations present unique challenges because harmful interactions often unfold through context-dependent, multi-turn exchanges, while victims may lack timely support during moments of harassment. In this study, we investigate how large language models (LLMs) can support both the detection of and response to online harassment in private messaging. Using a dataset of 80,053 Instagram direct messages donated by 26 adolescents aged 12-18, including youth with suicide risk factors, we first construct a human-labeled dataset of online harassment in private conversations and develop a context-aware cascading LLM classification pipeline. The proposed pipeline outperforms baseline toxicity classifiers trained primarily on public social media data. We then develop a victim-centered response framework that produces context-sensitive and psychologically-grounded AI-generated responses to online harassment messages. Human evaluators perceived the AI-generated responses as significantly more helpful than the original participant responses (95% CI: 0.767--0.815, p < .001), particularly in terms of emotional support and de-escalation. Our findings highlight the potential of context-aware and victim-centered AI systems to provide just-in-time support during harassment in private messaging environments. |
| title | Context-Aware Detection and Victim-Centered Response Generation for Online Harassment in Private Messaging |
| topic | Social and Information Networks Computation and Language Computers and Society 68T42 |
| url | https://arxiv.org/abs/2512.14700 |