Cover Image

More is Less: The Pitfalls of Multi-Model Synthetic Preference Data in DPO Safety Alignment

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Yifan, Chen, Runjin, Li, Bolian, Cho, David, Deng, Yihe, Zhang, Ruqi, Chen, Tianlong, Wang, Zhangyang, Grama, Ananth, Hong, Junyuan
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Cascade Reward Sampling for Efficient Decoding-Time Alignment
by: Li, Bolian, et al.
Published: (2024)

LLMs Can Get "Brain Rot": A Pilot Study on Twitter/X
by: Xing, Shuo, et al.
Published: (2025)

SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology
by: Wang, Yifan, et al.
Published: (2026)

Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
by: Li, Bolian, et al.
Published: (2026)

DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning
by: Wang, Yifan, et al.
Published: (2025)

LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
by: Perin, Gabriel J., et al.
Published: (2025)

SEAL: Steerable Reasoning Calibration of Large Language Models for Free
by: Chen, Runjin, et al.
Published: (2025)

From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications
by: Jaiswal, Ajay, et al.
Published: (2024)

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
by: Ding, Yi, et al.
Published: (2024)

Inference-Time Code Selection via Symbolic Equivalence Partitioning
by: Cho, David, et al.
Published: (2026)

Less is More: Improving LLM Alignment via Preference Data Selection
by: Deng, Xun, et al.
Published: (2025)

Entropy-MCMC: Sampling from Flat Basins with Ease
by: Li, Bolian, et al.
Published: (2023)

Making Reliable and Flexible Decisions in Long-tailed Classification
by: Li, Bolian, et al.
Published: (2025)

Deconvolving Complex Neuronal Networks into Interpretable Task-Specific Connectomes
by: Wang, Yifan, et al.
Published: (2024)

Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning
by: Deng, Yihe, et al.
Published: (2024)

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
by: Lochab, Anamika, et al.
Published: (2026)

Bayesian Computation in Deep Learning
by: Chen, Wenlong, et al.
Published: (2025)

CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignment
by: Liu, Jilong, et al.
Published: (2026)

Extracting and Understanding the Superficial Knowledge in Alignment
by: Chen, Runjin, et al.
Published: (2025)

Less is More: On Copy Complexity in Quantum Cryptography
by: Ananth, Prabhanjan, et al.
Published: (2025)

MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models
by: Pandit, Shrey, et al.
Published: (2025)

Robust Multi-Objective Preference Alignment with Online DPO
by: Gupta, Raghav, et al.
Published: (2025)

VideoDPO: Omni-Preference Alignment for Video Diffusion Generation
by: Liu, Runtao, et al.
Published: (2024)

LLaGA: Large Language and Graph Assistant
by: Chen, Runjin, et al.
Published: (2024)

Template-as-Ontology: Configurable Synthetic Data Infrastructure for Cross-Domain Manufacturing AI Validation
by: Chethan, Grama
Published: (2026)

Learning Self-Correction in Vision-Language Models via Rollout Augmentation
by: Ding, Yi, et al.
Published: (2026)

Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
by: Li, Bolian, et al.
Published: (2025)

PIPA: Preference Alignment as Prior-Informed Statistical Estimation
by: Li, Junbo, et al.
Published: (2025)

Shake to Leak: Fine-tuning Diffusion Models Can Amplify the Generative Privacy Risk
by: Li, Zhangheng, et al.
Published: (2024)

Less is More: Adaptive Coverage for Synthetic Training Data
by: Tavakkol, Sasan, et al.
Published: (2025)

Cat-DPO: Category-Adaptive Safety Alignment
by: Yang, Tiankai, et al.
Published: (2026)

MixDPO: Modeling Preference Strength for Pluralistic Alignment
by: Imai, Saki, et al.
Published: (2026)

Robust Online Classification: From Estimation to Denoising
by: Wu, Changlong, et al.
Published: (2023)

No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models
by: Wu, Changlong, et al.
Published: (2024)

Generalized Learning of Coefficients in Spectral Graph Convolutional Networks
by: Coşkun, Mustafa, et al.
Published: (2024)

Unified Multi-Modal Interactive & Reactive 3D Motion Generation via Rectified Flow
by: Gupta, Prerit, et al.
Published: (2025)

DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
by: Shang, Shuyao, et al.
Published: (2025)

Not All Synthetic Data Is Yours to Learn From
by: Alemohammad, Sina, et al.
Published: (2026)

CATNIP: LLM Unlearning via Calibrated and Tokenized Negative Preference Alignment
by: Yang, Zhengbang, et al.
Published: (2026)

Course-Correction: Safety Alignment Using Synthetic Preferences
by: Xu, Rongwu, et al.
Published: (2024)