Cover Image

When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loop

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Yang, Wei, Xiukun, Zhang, Xueru
Format:	Preprint
Published:	2026
Subjects:	Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Self-Consuming Generative Models with Adversarially Curated Data
by: Wei, Xiukun, et al.
Published: (2025)

Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions
by: Xie, Tian, et al.
Published: (2024)

Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences
by: Ferbach, Damien, et al.
Published: (2024)

Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
by: Zhou, Zhanhui, et al.
Published: (2024)

How Strategic Agents Respond: Comparing Analytical Models with LLM-Generated Responses in Strategic Classification
by: Xie, Tian, et al.
Published: (2025)

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
by: Zhang, Shenao, et al.
Published: (2024)

Market Games for Generative Models: Equilibria, Welfare, and Strategic Entry
by: Wei, Xiukun, et al.
Published: (2026)

Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy
by: Liu, Chris Yuhao, et al.
Published: (2025)

Non-linear Welfare-Aware Strategic Learning
by: Xie, Tian, et al.
Published: (2024)

Self-Play Preference Optimization for Language Model Alignment
by: Wu, Yue, et al.
Published: (2024)

When Incentives Backfire, Data Stops Being Human
by: Santy, Sebastin, et al.
Published: (2025)

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
by: Wang, Haoxiang, et al.
Published: (2024)

Self-Correcting Self-Consuming Loops for Generative Model Training
by: Gillman, Nate, et al.
Published: (2024)

Latent Embedding Adaptation for Human Preference Alignment in Diffusion Planners
by: Ng, Wen Zheng Terence, et al.
Published: (2025)

Human Alignment of Large Language Models through Online Preference Optimisation
by: Calandriello, Daniele, et al.
Published: (2024)

Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop
by: Wang, Yaxuan, et al.
Published: (2026)

Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking
by: Ren, Jie, et al.
Published: (2025)

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization
by: Zhou, Zhanhui, et al.
Published: (2023)

Towards Theoretical Understandings of Self-Consuming Generative Models
by: Fu, Shi, et al.
Published: (2024)

Adversarial Preference Learning for Robust LLM Alignment
by: Wang, Yuanfu, et al.
Published: (2025)

The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation
by: Falahati, Ali, et al.
Published: (2025)

Multi-Metric Preference Alignment for Generative Speech Restoration
by: Zhang, Junan, et al.
Published: (2025)

From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
by: Zeng, Siliang, et al.
Published: (2025)

Comparative Explanations: Explanation Guided Decision Making for Human-in-the-Loop Preference Selection
by: Chakraborty, Tanmay, et al.
Published: (2025)

Property-driven Protein Inverse Folding With Multi-Objective Preference Alignment
by: Hou, Xiaoyang, et al.
Published: (2026)

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
by: Zhang, Shenao, et al.
Published: (2024)

Risk-aware Direct Preference Optimization under Nested Risk Measure
by: Zhang, Lijun, et al.
Published: (2025)

Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models
by: Wang, Fei, et al.
Published: (2024)

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
by: Xiong, Wei, et al.
Published: (2023)

SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignment
by: Zhu, Wenqiao, et al.
Published: (2025)

A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops
by: Fu, Shi, et al.
Published: (2025)

Preference-Based Alignment of Discrete Diffusion Models
by: Borso, Umberto, et al.
Published: (2025)

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
by: Gupta, Taneesh, et al.
Published: (2025)

Self-Supervised Visual Preference Alignment
by: Zhu, Ke, et al.
Published: (2024)

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
by: Yang, Rui, et al.
Published: (2024)

Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuning
by: Liu, Guozhi, et al.
Published: (2025)

Enabling Self-Improving Agents to Learn at Test Time With Human-In-The-Loop Guidance
by: He, Yufei, et al.
Published: (2025)

PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model
by: Lin, Baijiong, et al.
Published: (2025)

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
by: Zhang, Yifan, et al.
Published: (2024)

Teaching Your Models to Understand Code via Focal Preference Alignment
by: Wu, Jie, et al.
Published: (2025)