STAR-1: Safer Alignment of Reasoning LLMs with 1K Data
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915611376877568 |
|---|---|
| author | Wang, Zijun Tu, Haoqin Wang, Yuhan Wu, Juncheng Liu, Yanqing Mei, Jieru Bartoldson, Brian R. Kailkhura, Bhavya Xie, Cihang |
| author_facet | Wang, Zijun Tu, Haoqin Wang, Yuhan Wu, Juncheng Liu, Yanqing Mei, Jieru Bartoldson, Brian R. Kailkhura, Bhavya Xie, Cihang |
| contents | This paper introduces STAR-1, a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) like DeepSeek-R1. Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs. Specifically, we begin by integrating existing open-source safety datasets from diverse sources. Then, we curate safety policies to generate policy-grounded deliberative reasoning samples. Lastly, we apply a GPT-4o-based safety scoring system to select training examples aligned with best practices. Experimental results show that fine-tuning LRMs with STAR-1 leads to an average 40% improvement in safety performance across four benchmarks, while only incurring a marginal decrease (e.g., an average of 1.1%) in reasoning ability measured across five reasoning tasks. Extensive ablation studies further validate the importance of our design principles in constructing STAR-1 and analyze its efficacy across both LRMs and traditional LLMs. Our project page is https://ucsc-vlaa.github.io/STAR-1. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01903 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | STAR-1: Safer Alignment of Reasoning LLMs with 1K Data Wang, Zijun Tu, Haoqin Wang, Yuhan Wu, Juncheng Liu, Yanqing Mei, Jieru Bartoldson, Brian R. Kailkhura, Bhavya Xie, Cihang Computation and Language Artificial Intelligence This paper introduces STAR-1, a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) like DeepSeek-R1. Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs. Specifically, we begin by integrating existing open-source safety datasets from diverse sources. Then, we curate safety policies to generate policy-grounded deliberative reasoning samples. Lastly, we apply a GPT-4o-based safety scoring system to select training examples aligned with best practices. Experimental results show that fine-tuning LRMs with STAR-1 leads to an average 40% improvement in safety performance across four benchmarks, while only incurring a marginal decrease (e.g., an average of 1.1%) in reasoning ability measured across five reasoning tasks. Extensive ablation studies further validate the importance of our design principles in constructing STAR-1 and analyze its efficacy across both LRMs and traditional LLMs. Our project page is https://ucsc-vlaa.github.io/STAR-1. |
| title | STAR-1: Safer Alignment of Reasoning LLMs with 1K Data |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.01903 |