Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.12804 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911268856659968 |
|---|---|
| author | Falahati, Ali Amiri, Mohammad Mohammadi Larson, Kate Golab, Lukasz |
| author_facet | Falahati, Ali Amiri, Mohammad Mohammadi Larson, Kate Golab, Lukasz |
| contents | In self-consuming generative models that train on their own outputs, alignment with user preferences becomes a recursive rather than one-time process. We provide the first formal foundation for analyzing the long-term effects of such recursive retraining on alignment. Under a two-stage curation mechanism based on the Bradley-Terry (BT) model, we model alignment as an interaction between two factions: the Model Owner, who filters which outputs should be learned by the model, and the Public User, who determines which outputs are ultimately shared and retained through interactions with the model. Our analysis reveals three structural convergence regimes depending on the degree of preference alignment: consensus collapse, compromise on shared optima, and asymmetric refinement. We prove a fundamental impossibility theorem: no recursive BT-based curation mechanism can simultaneously preserve diversity, ensure symmetric influence, and eliminate dependence on initialization. Framing the process as dynamic social choice, we show that alignment is not a static goal but an evolving equilibrium, shaped both by power asymmetries and path dependence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12804 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation Falahati, Ali Amiri, Mohammad Mohammadi Larson, Kate Golab, Lukasz Machine Learning Artificial Intelligence In self-consuming generative models that train on their own outputs, alignment with user preferences becomes a recursive rather than one-time process. We provide the first formal foundation for analyzing the long-term effects of such recursive retraining on alignment. Under a two-stage curation mechanism based on the Bradley-Terry (BT) model, we model alignment as an interaction between two factions: the Model Owner, who filters which outputs should be learned by the model, and the Public User, who determines which outputs are ultimately shared and retained through interactions with the model. Our analysis reveals three structural convergence regimes depending on the degree of preference alignment: consensus collapse, compromise on shared optima, and asymmetric refinement. We prove a fundamental impossibility theorem: no recursive BT-based curation mechanism can simultaneously preserve diversity, ensure symmetric influence, and eliminate dependence on initialization. Framing the process as dynamic social choice, we show that alignment is not a static goal but an evolving equilibrium, shaped both by power asymmetries and path dependence. |
| title | The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2511.12804 |