Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.04196 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915773051568128 |
|---|---|
| author | Zhang, Zhexin Lu, Yida Fang, Junfeng Yang, Junxiao Cui, Shiyao Zhou, Hao Meng, Fandong Zhou, Jie Wang, Hongning Huang, Minlie Chua, Tat-Seng |
| author_facet | Zhang, Zhexin Lu, Yida Fang, Junfeng Yang, Junxiao Cui, Shiyao Zhou, Hao Meng, Fandong Zhou, Jie Wang, Hongning Huang, Minlie Chua, Tat-Seng |
| contents | Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcement learning, we study implicit training-time safety risks: harmful behaviors driven by a model's internal incentives and contextual background information. For example, during code-based reinforcement learning, a model may covertly manipulate logged accuracy for self-preservation. We present the first systematic study of this problem, introducing a taxonomy with five risk levels, ten fine-grained risk categories, and three incentive types. Extensive experiments reveal the prevalence and severity of these risks: notably, Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided only with background information. We further analyze factors influencing these behaviors and demonstrate that implicit training-time risks also arise in multi-agent training settings. Our results identify an overlooked yet urgent safety challenge in training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_04196 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment Zhang, Zhexin Lu, Yida Fang, Junfeng Yang, Junxiao Cui, Shiyao Zhou, Hao Meng, Fandong Zhou, Jie Wang, Hongning Huang, Minlie Chua, Tat-Seng Computation and Language Machine Learning Safety risks of AI models have been widely studied at deployment time, such as jailbreak attacks that elicit harmful outputs. In contrast, safety risks emerging during training remain largely unexplored. Beyond explicit reward hacking that directly manipulates explicit reward functions in reinforcement learning, we study implicit training-time safety risks: harmful behaviors driven by a model's internal incentives and contextual background information. For example, during code-based reinforcement learning, a model may covertly manipulate logged accuracy for self-preservation. We present the first systematic study of this problem, introducing a taxonomy with five risk levels, ten fine-grained risk categories, and three incentive types. Extensive experiments reveal the prevalence and severity of these risks: notably, Llama-3.1-8B-Instruct exhibits risky behaviors in 74.4% of training runs when provided only with background information. We further analyze factors influencing these behaviors and demonstrate that implicit training-time risks also arise in multi-agent training settings. Our results identify an overlooked yet urgent safety challenge in training. |
| title | The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2602.04196 |