Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929636688003072 |
|---|---|
| author | Kumar, Shanu Kholkar, Gauri Mendke, Saish Sadana, Anubhav Agrawal, Parag Dandapat, Sandipan |
| author_facet | Kumar, Shanu Kholkar, Gauri Mendke, Saish Sadana, Anubhav Agrawal, Parag Dandapat, Sandipan |
| contents | With the growth of social media and large language models, content moderation has become crucial. Many existing datasets lack adequate representation of different groups, resulting in unreliable assessments. To tackle this, we propose a socio-culturally aware evaluation framework for LLM-driven content moderation and introduce a scalable method for creating diverse datasets using persona-based generation. Our analysis reveals that these datasets provide broader perspectives and pose greater challenges for LLMs than diversity-focused generation methods without personas. This challenge is especially pronounced in smaller LLMs, emphasizing the difficulties they encounter in moderating such diverse content. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13578 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation Kumar, Shanu Kholkar, Gauri Mendke, Saish Sadana, Anubhav Agrawal, Parag Dandapat, Sandipan Computation and Language Artificial Intelligence With the growth of social media and large language models, content moderation has become crucial. Many existing datasets lack adequate representation of different groups, resulting in unreliable assessments. To tackle this, we propose a socio-culturally aware evaluation framework for LLM-driven content moderation and introduce a scalable method for creating diverse datasets using persona-based generation. Our analysis reveals that these datasets provide broader perspectives and pose greater challenges for LLMs than diversity-focused generation methods without personas. This challenge is especially pronounced in smaller LLMs, emphasizing the difficulties they encounter in moderating such diverse content. |
| title | Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.13578 |