Safety Pretraining: Toward the Next Generation of Safe AI
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918140931211264 |
|---|---|
| author | Maini, Pratyush Goyal, Sachin Sam, Dylan Robey, Alex Savani, Yash Jiang, Yiding Zou, Andy Fredrikson, Matt Lipton, Zacharcy C. Kolter, J. Zico |
| author_facet | Maini, Pratyush Goyal, Sachin Sam, Dylan Robey, Alex Savani, Yash Jiang, Yiding Zou, Andy Fredrikson, Matt Lipton, Zacharcy C. Kolter, J. Zico |
| contents | As large language models (LLMs) are increasingly deployed in high-stakes settings, the risk of generating harmful or toxic content remains a central challenge. Post-hoc alignment methods are brittle: once unsafe patterns are learned during pretraining, they are hard to remove. In this work, we present a data-centric pretraining framework that builds safety into the model from the start. Our framework consists of four key steps: (i) Safety Filtering: building a safety classifier to classify webdata into safe and unsafe categories; (ii) Safety Rephrasing: we recontextualize unsafe webdata into safer narratives; (iii) Native Refusal: we develop RefuseWeb and Moral Education pretraining datasets that actively teach model to refuse on unsafe content and the moral reasoning behind it, and (iv) Harmfulness-Tag annotated pretraining: we flag unsafe content during pretraining using a special token, and use it to steer model away from unsafe generations at inference. Our safety-pretrained models reduce attack success rates from 38.8\% to 8.4\% on standard LLM safety benchmarks with no performance degradation on general tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16980 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Safety Pretraining: Toward the Next Generation of Safe AI Maini, Pratyush Goyal, Sachin Sam, Dylan Robey, Alex Savani, Yash Jiang, Yiding Zou, Andy Fredrikson, Matt Lipton, Zacharcy C. Kolter, J. Zico Machine Learning As large language models (LLMs) are increasingly deployed in high-stakes settings, the risk of generating harmful or toxic content remains a central challenge. Post-hoc alignment methods are brittle: once unsafe patterns are learned during pretraining, they are hard to remove. In this work, we present a data-centric pretraining framework that builds safety into the model from the start. Our framework consists of four key steps: (i) Safety Filtering: building a safety classifier to classify webdata into safe and unsafe categories; (ii) Safety Rephrasing: we recontextualize unsafe webdata into safer narratives; (iii) Native Refusal: we develop RefuseWeb and Moral Education pretraining datasets that actively teach model to refuse on unsafe content and the moral reasoning behind it, and (iv) Harmfulness-Tag annotated pretraining: we flag unsafe content during pretraining using a special token, and use it to steer model away from unsafe generations at inference. Our safety-pretrained models reduce attack success rates from 38.8\% to 8.4\% on standard LLM safety benchmarks with no performance degradation on general tasks. |
| title | Safety Pretraining: Toward the Next Generation of Safe AI |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.16980 |