How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915943898152960 |
|---|---|
| author | Zhang, Zhexin Loye, Xian Qi Huang, Victor Shea-Jay Yang, Junxiao Zhu, Qi Cui, Shiyao Mi, Fei Shang, Lifeng Wang, Yingkang Wang, Hongning Huang, Minlie |
| author_facet | Zhang, Zhexin Loye, Xian Qi Huang, Victor Shea-Jay Yang, Junxiao Zhu, Qi Cui, Shiyao Mi, Fei Shang, Lifeng Wang, Yingkang Wang, Hongning Huang, Minlie |
| contents | Large Reasoning Models (LRMs) have achieved remarkable success on reasoning-intensive tasks such as mathematics and programming. However, their enhanced reasoning capabilities do not necessarily translate to improved safety performance-and in some cases, may even degrade it. This raises an important research question: how should we enhance the safety of LRMs? In this paper, we present a comprehensive empirical study on how to enhance the safety of LRMs through Supervised Fine-Tuning (SFT). Our investigation begins with an unexpected observation: directly distilling safe responses from DeepSeek-R1 fails to significantly enhance safety. We analyze this phenomenon and identify five key risky patterns that contribute to it. We then demonstrate that explicitly addressing these issues during the data distillation process can lead to substantial safety improvements. Next, we explore whether a long and complex reasoning process is necessary for achieving safety. Interestingly, we find that simply using short or template-based reasoning process can attain comparable safety performance. These findings prompt a deeper reflection on the role of reasoning in ensuring safety. Finally, we conduct a comprehensive ablation study to reveal the impact of different training configurations. Overall, we hope our empirical study could provide a more holistic picture on enhancing the safety of LRMs. The code and data used in our experiments are released in https://github.com/thu-coai/LRM-Safety-Study. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15404 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study Zhang, Zhexin Loye, Xian Qi Huang, Victor Shea-Jay Yang, Junxiao Zhu, Qi Cui, Shiyao Mi, Fei Shang, Lifeng Wang, Yingkang Wang, Hongning Huang, Minlie Computation and Language Large Reasoning Models (LRMs) have achieved remarkable success on reasoning-intensive tasks such as mathematics and programming. However, their enhanced reasoning capabilities do not necessarily translate to improved safety performance-and in some cases, may even degrade it. This raises an important research question: how should we enhance the safety of LRMs? In this paper, we present a comprehensive empirical study on how to enhance the safety of LRMs through Supervised Fine-Tuning (SFT). Our investigation begins with an unexpected observation: directly distilling safe responses from DeepSeek-R1 fails to significantly enhance safety. We analyze this phenomenon and identify five key risky patterns that contribute to it. We then demonstrate that explicitly addressing these issues during the data distillation process can lead to substantial safety improvements. Next, we explore whether a long and complex reasoning process is necessary for achieving safety. Interestingly, we find that simply using short or template-based reasoning process can attain comparable safety performance. These findings prompt a deeper reflection on the role of reasoning in ensuring safety. Finally, we conduct a comprehensive ablation study to reveal the impact of different training configurations. Overall, we hope our empirical study could provide a more holistic picture on enhancing the safety of LRMs. The code and data used in our experiments are released in https://github.com/thu-coai/LRM-Safety-Study. |
| title | How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.15404 |