Image de couverture de livre

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yang, Zhiqin, Zhang, Yonggang, Xue, Wei, Fang, Dong, Han, Bo, Guo, Yike
Format:	Preprint
Publié:	2026
Sujets:	Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
par: Chen, Ziyi, et autres
Publié: (2025)

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
par: Liu, Zhihan, et autres
Publié: (2024)

DPO Meets PPO: Reinforced Token Optimization for RLHF
par: Zhong, Han, et autres
Publié: (2024)

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
par: Zhou, Xingyu, et autres
Publié: (2025)

MemFly: On-the-Fly Memory Optimization via Information Bottleneck
par: Zhang, Zhenyuan, et autres
Publié: (2026)

Greedy Sampling Is Provably Efficient for RLHF
par: Wu, Di, et autres
Publié: (2025)

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
par: Pal, Arka, et autres
Publié: (2024)

ClawNet: Human-Symbiotic Agent Network for Cross-User Autonomous Cooperation
par: Yang, Zhiqin, et autres
Publié: (2026)

RLHF Workflow: From Reward Modeling to Online RLHF
par: Dong, Hanze, et autres
Publié: (2024)

FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clients
par: Wen, Tian, et autres
Publié: (2026)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
par: Wang, Bo, et autres
Publié: (2025)

Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
par: Zhu, Yu, et autres
Publié: (2024)

Towards Reliable Alignment: Uncertainty-aware RLHF
par: Banerjee, Debangshu, et autres
Publié: (2024)

Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback
par: Ji, Jiaming, et autres
Publié: (2025)

Provable Acceleration for Diffusion Models under Minimal Assumptions
par: Li, Gen, et autres
Publié: (2024)

The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment
par: Pollanen, Marco
Publié: (2026)

Boosting Deductive Reasoning with Step Signals In RLHF
par: Li, Jialian, et autres
Publié: (2024)

Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework
par: Kim, Kihyun, et autres
Publié: (2025)

Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
par: Feuer, Benjamin, et autres
Publié: (2024)

LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment
par: Li, Shipeng, et autres
Publié: (2025)

Provable Failure of Language Models in Learning Majority Boolean Logic via Gradient Descent
par: Chen, Bo, et autres
Publié: (2025)

ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
par: Lin, Xiaoqiang, et autres
Publié: (2025)

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
par: Zhang, Zhengze, et autres
Publié: (2025)

Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
par: Qi, Xuan, et autres
Publié: (2025)

Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
par: Lyu, Kaifeng, et autres
Publié: (2023)

MixDPO: Modeling Preference Strength for Pluralistic Alignment
par: Imai, Saki, et autres
Publié: (2026)

APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
par: Srewa, Mahmoud, et autres
Publié: (2026)

MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems
par: Cai, Qianshu, et autres
Publié: (2026)

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
par: Xiong, Wei, et autres
Publié: (2023)

VideoDPO: Omni-Preference Alignment for Video Diffusion Generation
par: Liu, Runtao, et autres
Publié: (2024)

Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy
par: Kumarappan, Adarsh, et autres
Publié: (2026)

PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
par: Ji, Jiaming, et autres
Publié: (2024)

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
par: Hu, Jian, et autres
Publié: (2024)

ROCM: RLHF on consistency models
par: Shekhar, Shivanshu, et autres
Publié: (2025)

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
par: Xie, Tengyang, et autres
Publié: (2024)

MaxMin-RLHF: Alignment with Diverse Human Preferences
par: Chakraborty, Souradip, et autres
Publié: (2024)

Democratic Preference Alignment via Sortition-Weighted RLHF
par: Sana, Suvadip, et autres
Publié: (2026)

Policy Filtration for RLHF to Mitigate Noise in Reward Models
par: Zhang, Chuheng, et autres
Publié: (2024)

Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning
par: Zhao, Weiye, et autres
Publié: (2024)

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization
par: He, Chaoyue, et autres
Publié: (2026)