Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917268722548736 |
|---|---|
| author | Han, Siwei Xiong, Kaiwen Liu, Jiaqi Ye, Xinyu Su, Yaofeng Duan, Wenbo Liu, Xinyuan Xie, Cihang Bansal, Mohit Ding, Mingyu Zhang, Linjun Yao, Huaxiu |
| author_facet | Han, Siwei Xiong, Kaiwen Liu, Jiaqi Ye, Xinyu Su, Yaofeng Duan, Wenbo Liu, Xinyuan Xie, Cihang Bansal, Mohit Ding, Mingyu Zhang, Linjun Yao, Huaxiu |
| contents | As Large Language Model (LLM) agents increasingly gain self-evolutionary capabilities to adapt and refine their strategies through real-world interaction, their long-term reliability becomes a critical concern. We identify the Alignment Tipping Process (ATP), a critical post-deployment risk unique to self-evolving LLM agents. Unlike training-time failures, ATP arises when continual interaction drives agents to abandon alignment constraints established during training in favor of reinforced, self-interested strategies. We formalize and analyze ATP through two complementary paradigms: Self-Interested Exploration, where repeated high-reward deviations induce individual behavioral drift, and Imitative Strategy Diffusion, where deviant behaviors spread across multi-agent systems. Building on these paradigms, we construct controllable testbeds and benchmark both open and closed-source LLMs. Our experiments show that alignment benefits erode rapidly under self-evolution, with initially aligned models converging toward unaligned states. In multi-agent settings, successful violations diffuse quickly, leading to collective misalignment. Moreover, current reinforcement learning-based alignment methods provide limited defenses against alignment tipping. These findings demonstrate that alignment of LLM agents is not a static property but a fragile and dynamic one, vulnerable to feedback-driven decay during deployment. Our data and code are available at https://github.com/aiming-lab/ATP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04860 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails Han, Siwei Xiong, Kaiwen Liu, Jiaqi Ye, Xinyu Su, Yaofeng Duan, Wenbo Liu, Xinyuan Xie, Cihang Bansal, Mohit Ding, Mingyu Zhang, Linjun Yao, Huaxiu Machine Learning Artificial Intelligence As Large Language Model (LLM) agents increasingly gain self-evolutionary capabilities to adapt and refine their strategies through real-world interaction, their long-term reliability becomes a critical concern. We identify the Alignment Tipping Process (ATP), a critical post-deployment risk unique to self-evolving LLM agents. Unlike training-time failures, ATP arises when continual interaction drives agents to abandon alignment constraints established during training in favor of reinforced, self-interested strategies. We formalize and analyze ATP through two complementary paradigms: Self-Interested Exploration, where repeated high-reward deviations induce individual behavioral drift, and Imitative Strategy Diffusion, where deviant behaviors spread across multi-agent systems. Building on these paradigms, we construct controllable testbeds and benchmark both open and closed-source LLMs. Our experiments show that alignment benefits erode rapidly under self-evolution, with initially aligned models converging toward unaligned states. In multi-agent settings, successful violations diffuse quickly, leading to collective misalignment. Moreover, current reinforcement learning-based alignment methods provide limited defenses against alignment tipping. These findings demonstrate that alignment of LLM agents is not a static property but a fragile and dynamic one, vulnerable to feedback-driven decay during deployment. Our data and code are available at https://github.com/aiming-lab/ATP. |
| title | Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.04860 |