SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913861862424576 |
|---|---|
| author | Li, Jing-Jing Pyatkin, Valentina Kleiman-Weiner, Max Jiang, Liwei Dziri, Nouha Collins, Anne G. E. Borg, Jana Schaich Sap, Maarten Choi, Yejin Levine, Sydney |
| author_facet | Li, Jing-Jing Pyatkin, Valentina Kleiman-Weiner, Max Jiang, Liwei Dziri, Nouha Collins, Anne G. E. Borg, Jana Schaich Sap, Maarten Choi, Yejin Levine, Sydney |
| contents | The ideal AI safety moderation system would be both structurally interpretable (so its decisions can be reliably explained) and steerable (to align to safety standards and reflect a community's values), which current systems fall short on. To address this gap, we present SafetyAnalyst, a novel AI safety moderation framework. Given an AI behavior, SafetyAnalyst uses chain-of-thought reasoning to analyze its potential consequences by creating a structured "harm-benefit tree," which enumerates harmful and beneficial actions and effects the AI behavior may lead to, along with likelihood, severity, and immediacy labels that describe potential impacts on stakeholders. SafetyAnalyst then aggregates all effects into a harmfulness score using 28 fully interpretable weight parameters, which can be aligned to particular safety preferences. We applied this framework to develop an open-source LLM prompt safety classification system, distilled from 18.5 million harm-benefit features generated by frontier LLMs on 19k prompts. On comprehensive benchmarks, we show that SafetyAnalyst (average F1=0.81) outperforms existing moderation systems (average F1$<$0.72) on prompt safety classification, while offering the additional advantages of interpretability, transparency, and steerability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_16665 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior Li, Jing-Jing Pyatkin, Valentina Kleiman-Weiner, Max Jiang, Liwei Dziri, Nouha Collins, Anne G. E. Borg, Jana Schaich Sap, Maarten Choi, Yejin Levine, Sydney Computation and Language Computers and Society The ideal AI safety moderation system would be both structurally interpretable (so its decisions can be reliably explained) and steerable (to align to safety standards and reflect a community's values), which current systems fall short on. To address this gap, we present SafetyAnalyst, a novel AI safety moderation framework. Given an AI behavior, SafetyAnalyst uses chain-of-thought reasoning to analyze its potential consequences by creating a structured "harm-benefit tree," which enumerates harmful and beneficial actions and effects the AI behavior may lead to, along with likelihood, severity, and immediacy labels that describe potential impacts on stakeholders. SafetyAnalyst then aggregates all effects into a harmfulness score using 28 fully interpretable weight parameters, which can be aligned to particular safety preferences. We applied this framework to develop an open-source LLM prompt safety classification system, distilled from 18.5 million harm-benefit features generated by frontier LLMs on 19k prompts. On comprehensive benchmarks, we show that SafetyAnalyst (average F1=0.81) outperforms existing moderation systems (average F1$<$0.72) on prompt safety classification, while offering the additional advantages of interpretability, transparency, and steerability. |
| title | SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2410.16665 |