Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.06550 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910865424384000 |
|---|---|
| author | Yin, Fan Laban, Philippe Peng, Xiangyu Zhou, Yilun Mao, Yixin Vats, Vaibhav Ross, Linnea Agarwal, Divyansh Xiong, Caiming Wu, Chien-Sheng |
| author_facet | Yin, Fan Laban, Philippe Peng, Xiangyu Zhou, Yilun Mao, Yixin Vats, Vaibhav Ross, Linnea Agarwal, Divyansh Xiong, Caiming Wu, Chien-Sheng |
| contents | Malicious content generated by large language models (LLMs) can pose varying degrees of harm. Although existing LLM-based moderators can detect harmful content, they struggle to assess risk levels and may miss lower-risk outputs. Accurate risk assessment allows platforms with different safety thresholds to tailor content filtering and rejection. In this paper, we introduce per-topic severity rubrics for 11 harmful topics and build BingoGuard, an LLM-based moderation system designed to predict both binary safety labels and severity levels. To address the lack of annotations on levels of severity, we propose a scalable generate-then-filter framework that first generates responses across different severity levels and then filters out low-quality responses. Using this framework, we create BingoGuardTrain, a training dataset with 54,897 examples covering a variety of topics, response severity, styles, and BingoGuardTest, a test set with 988 examples explicitly labeled based on our severity rubrics that enables fine-grained analysis on model behaviors on different severity levels. Our BingoGuard-8B, trained on BingoGuardTrain, achieves the state-of-the-art performance on several moderation benchmarks, including WildGuardTest and HarmBench, as well as BingoGuardTest, outperforming best public models, WildGuard, by 4.3\%. Our analysis demonstrates that incorporating severity levels into training significantly enhances detection performance and enables the model to effectively gauge the severity of harmful responses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06550 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BingoGuard: LLM Content Moderation Tools with Risk Levels Yin, Fan Laban, Philippe Peng, Xiangyu Zhou, Yilun Mao, Yixin Vats, Vaibhav Ross, Linnea Agarwal, Divyansh Xiong, Caiming Wu, Chien-Sheng Computation and Language Malicious content generated by large language models (LLMs) can pose varying degrees of harm. Although existing LLM-based moderators can detect harmful content, they struggle to assess risk levels and may miss lower-risk outputs. Accurate risk assessment allows platforms with different safety thresholds to tailor content filtering and rejection. In this paper, we introduce per-topic severity rubrics for 11 harmful topics and build BingoGuard, an LLM-based moderation system designed to predict both binary safety labels and severity levels. To address the lack of annotations on levels of severity, we propose a scalable generate-then-filter framework that first generates responses across different severity levels and then filters out low-quality responses. Using this framework, we create BingoGuardTrain, a training dataset with 54,897 examples covering a variety of topics, response severity, styles, and BingoGuardTest, a test set with 988 examples explicitly labeled based on our severity rubrics that enables fine-grained analysis on model behaviors on different severity levels. Our BingoGuard-8B, trained on BingoGuardTrain, achieves the state-of-the-art performance on several moderation benchmarks, including WildGuardTest and HarmBench, as well as BingoGuardTest, outperforming best public models, WildGuard, by 4.3\%. Our analysis demonstrates that incorporating severity levels into training significantly enhances detection performance and enables the model to effectively gauge the severity of harmful responses. |
| title | BingoGuard: LLM Content Moderation Tools with Risk Levels |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.06550 |