Cover Image

What Matters in Data for DPO?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Pan, Yu, Cai, Zhongze, Chen, Guanting, Zhong, Huaiyang, Wang, Chonghuan
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Optimistic Reinforcement Learning with Quantile Objectives
by: Alipour-Vaezi, Mohammad, et al.
Published: (2025)

OMGPT: A Sequence Modeling Framework for Data-driven Operational Decision Making
by: Wang, Hanzhao, et al.
Published: (2025)

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
by: Liu, Shang, et al.
Published: (2024)

Risk Profiling and Modulation for LLMs
by: Wang, Yikai, et al.
Published: (2025)

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
by: Zhang, Zhengze, et al.
Published: (2025)

Towards Better Statistical Understanding of Watermarking LLMs
by: Cai, Zhongze, et al.
Published: (2024)

Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Making
by: Wang, Hanzhao, et al.
Published: (2024)

Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
by: Chen, Ziyi, et al.
Published: (2025)

In-Context Curiosity: Distilling Exploration for Decision-Pretrained Transformers on Bandit Tasks
by: Yang, Huitao, et al.
Published: (2025)

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization
by: He, Chaoyue, et al.
Published: (2026)

DPO Meets PPO: Reinforced Token Optimization for RLHF
by: Zhong, Han, et al.
Published: (2024)

$β$-DPO: Direct Preference Optimization with Dynamic $β$
by: Wu, Junkang, et al.
Published: (2024)

Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs
by: Oh, Giyeong, et al.
Published: (2026)

$ξ$-DPO: Direct Preference Optimization via Ratio Reward Margin
by: Fan, Zhengyuan, et al.
Published: (2026)

Understanding the Impact of Sampling Quality in Direct Preference Optimization
by: Kim, Kyung Rok, et al.
Published: (2025)

The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment
by: Pollanen, Marco
Published: (2026)

SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
by: Kim, Geon-Hyeong, et al.
Published: (2025)

Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
by: Dong, Guanting, et al.
Published: (2024)

Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
by: Qi, Xuan, et al.
Published: (2025)

C2-DPO: Constrained Controlled Direct Preference Optimization
by: Asadi, Kavosh, et al.
Published: (2025)

Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification
by: Liu, Shang, et al.
Published: (2024)

ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
by: Lin, Xiaoqiang, et al.
Published: (2025)

Context and Diversity Matter: The Emergence of In-Context Learning in World Models
by: Wang, Fan, et al.
Published: (2025)

Margin Adaptive DPO: Leveraging Reward Model for Granular Control in Preference Optimization
by: Rho, Hyung Gyu
Published: (2025)

GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO
by: Zhao, Yiyang, et al.
Published: (2025)

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
by: Yang, Zhiqin, et al.
Published: (2026)

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
by: Wu, Junkang, et al.
Published: (2024)

Aligning Compound AI Systems via System-level DPO
by: Wang, Xiangwen, et al.
Published: (2025)

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
by: Zhou, Xingyu, et al.
Published: (2025)

MixDPO: Modeling Preference Strength for Pluralistic Alignment
by: Imai, Saki, et al.
Published: (2026)

Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
by: Lai, Xin, et al.
Published: (2024)

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering
by: Mohamed, Anas, et al.
Published: (2025)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
by: Wang, Bo, et al.
Published: (2025)

Labels Matter More Than Models: Rethinking the Unsupervised Paradigm in Time Series Anomaly Detection
by: Zhong, Zhijie, et al.
Published: (2025)

EnerBridge-DPO: Energy-Guided Protein Inverse Folding with Markov Bridges and Direct Preference Optimization
by: Rong, Dingyi, et al.
Published: (2025)

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
by: Pal, Arka, et al.
Published: (2024)

Graph Unlearning Meets Influence-aware Negative Preference Optimization
by: Chen, Qiang, et al.
Published: (2025)

What Matters for Model Merging at Scale?
by: Yadav, Prateek, et al.
Published: (2024)

VideoDPO: Omni-Preference Alignment for Video Diffusion Generation
by: Liu, Runtao, et al.
Published: (2024)

TRACE: Distilling Where It Matters via Token-Routed Self On-Policy Alignment
by: Wang, Jiaxuan, et al.
Published: (2026)