Capability-Oriented Training Induced Alignment Risk
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908830091182080 |
|---|---|
| author | Zhou, Yujun Huang, Yue Bao, Han Guo, Kehan Liang, Zhenwen Chen, Pin-Yu Gao, Tian Geyer, Werner Moniz, Nuno Chawla, Nitesh V Zhang, Xiangliang |
| author_facet | Zhou, Yujun Huang, Yue Bao, Han Guo, Kehan Liang, Zhenwen Chen, Pin-Yu Gao, Tian Geyer, Werner Moniz, Nuno Chawla, Nitesh V Zhang, Xiangliang |
| contents | While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk is emerging: capability-oriented training induced exploitation. We investigate whether language models, when trained with reinforcement learning (RL) in environments with implicit loopholes, will spontaneously learn to exploit these flaws to maximize their reward, even without any malicious intent in their training. To test this, we design a suite of four diverse "vulnerability games", each presenting a unique, exploitable flaw related to context-conditional compliance, proxy metrics, reward tampering, and self-evaluation. Our experiments show that models consistently learn to exploit these vulnerabilities, discovering opportunistic strategies that significantly increase their reward at the expense of task correctness or safety. More critically, we find that these exploitative strategies are not narrow "tricks" but generalizable skills; they can be transferred to new tasks and even "distilled" from a capable teacher model to other student models through data alone. Our findings reveal that capability-oriented training induced risks pose a fundamental challenge to current alignment approaches, suggesting that future AI safety work must extend beyond content moderation to rigorously auditing and securing the training environments and reward mechanisms themselves. Code is available at https://github.com/YujunZhou/Capability_Oriented_Alignment_Risk. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12124 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Capability-Oriented Training Induced Alignment Risk Zhou, Yujun Huang, Yue Bao, Han Guo, Kehan Liang, Zhenwen Chen, Pin-Yu Gao, Tian Geyer, Werner Moniz, Nuno Chawla, Nitesh V Zhang, Xiangliang Machine Learning Computation and Language While most AI alignment research focuses on preventing models from generating explicitly harmful content, a more subtle risk is emerging: capability-oriented training induced exploitation. We investigate whether language models, when trained with reinforcement learning (RL) in environments with implicit loopholes, will spontaneously learn to exploit these flaws to maximize their reward, even without any malicious intent in their training. To test this, we design a suite of four diverse "vulnerability games", each presenting a unique, exploitable flaw related to context-conditional compliance, proxy metrics, reward tampering, and self-evaluation. Our experiments show that models consistently learn to exploit these vulnerabilities, discovering opportunistic strategies that significantly increase their reward at the expense of task correctness or safety. More critically, we find that these exploitative strategies are not narrow "tricks" but generalizable skills; they can be transferred to new tasks and even "distilled" from a capable teacher model to other student models through data alone. Our findings reveal that capability-oriented training induced risks pose a fundamental challenge to current alignment approaches, suggesting that future AI safety work must extend beyond content moderation to rigorously auditing and securing the training environments and reward mechanisms themselves. Code is available at https://github.com/YujunZhou/Capability_Oriented_Alignment_Risk. |
| title | Capability-Oriented Training Induced Alignment Risk |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2602.12124 |