Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.05328 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912110940782592 |
|---|---|
| author | Ye, Teng Yan, Hanson Huang, Xuhuan Grogan, Connor Yuan, Walter Mei, Qiaozhu Jackson, Matthew O. |
| author_facet | Ye, Teng Yan, Hanson Huang, Xuhuan Grogan, Connor Yuan, Walter Mei, Qiaozhu Jackson, Matthew O. |
| contents | With the rise of social media and peer-to-peer networks, users increasingly rely on crowdsourced responses for information and assistance. However, the mechanisms used to rank and promote responses often prioritize and end up biasing in favor of timeliness over quality, which may result in suboptimal support for help-seekers. We analyze millions of responses to mental health-related posts, utilizing large language models (LLMs) to assess the multi-dimensional quality of content, including relevance, empathy, and cultural alignment, among other aspects. Our findings reveal a mismatch between content quality and attention allocation: earlier responses - despite being relatively lower in quality - receive disproportionately high fractions of upvotes and visibility due to platform ranking algorithms. We demonstrate that the quality of the top-ranked responses could be improved by up to 39 percent, and even the simplest re-ranking strategy could significantly improve the quality of top responses, highlighting the need for more nuanced ranking mechanisms that prioritize both timeliness and content quality, especially emotional engagement in online mental health communities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05328 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Content Quality vs. Attention Allocation: An LLM-Based Case Study in Peer-to-peer Mental Health Networks Ye, Teng Yan, Hanson Huang, Xuhuan Grogan, Connor Yuan, Walter Mei, Qiaozhu Jackson, Matthew O. Social and Information Networks 91D30, 94A16 With the rise of social media and peer-to-peer networks, users increasingly rely on crowdsourced responses for information and assistance. However, the mechanisms used to rank and promote responses often prioritize and end up biasing in favor of timeliness over quality, which may result in suboptimal support for help-seekers. We analyze millions of responses to mental health-related posts, utilizing large language models (LLMs) to assess the multi-dimensional quality of content, including relevance, empathy, and cultural alignment, among other aspects. Our findings reveal a mismatch between content quality and attention allocation: earlier responses - despite being relatively lower in quality - receive disproportionately high fractions of upvotes and visibility due to platform ranking algorithms. We demonstrate that the quality of the top-ranked responses could be improved by up to 39 percent, and even the simplest re-ranking strategy could significantly improve the quality of top responses, highlighting the need for more nuanced ranking mechanisms that prioritize both timeliness and content quality, especially emotional engagement in online mental health communities. |
| title | Content Quality vs. Attention Allocation: An LLM-Based Case Study in Peer-to-peer Mental Health Networks |
| topic | Social and Information Networks 91D30, 94A16 |
| url | https://arxiv.org/abs/2411.05328 |