Cover Image

BPO: Revisiting Preference Modeling in Direct Preference Optimization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sun, Lin, Liu, Chuang, Liu, Peng, Li, Bingyang, Lu, Weijia, Wu, Ning
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment
by: Wang, Sizhe, et al.
Published: (2024)

Token-level Direct Preference Optimization
by: Zeng, Yongcheng, et al.
Published: (2024)

FreePRM: Training Process Reward Models Without Ground Truth Process Labels
by: Sun, Lin, et al.
Published: (2025)

Understanding Reference Policies in Direct Preference Optimization
by: Liu, Yixin, et al.
Published: (2024)

Enhancing LLM Safety via Constrained Direct Preference Optimization
by: Liu, Zixuan, et al.
Published: (2024)

Orthogonal Finetuning for Direct Preference Optimization
by: Yang, Chenxu, et al.
Published: (2024)

No Preference Left Behind: Group Distributional Preference Optimization
by: Yao, Binwei, et al.
Published: (2024)

Direct Judgement Preference Optimization
by: Wang, Peifeng, et al.
Published: (2024)

On Extending Direct Preference Optimization to Accommodate Ties
by: Chen, Jinghong, et al.
Published: (2024)

Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models
by: Xiang, Hao, et al.
Published: (2024)

Length Desensitization in Direct Preference Optimization
by: Liu, Wei, et al.
Published: (2024)

Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
by: Wang, Tianduo, et al.
Published: (2024)

Token-weighted Direct Preference Optimization with Attention
by: Huang, Chengyu, et al.
Published: (2026)

SDPO: Segment-Level Direct Preference Optimization for Social Agents
by: Kong, Aobo, et al.
Published: (2025)

New Desiderata for Direct Preference Optimization
by: Hu, Xiangkun, et al.
Published: (2024)

Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
by: Jiang, Yuxin, et al.
Published: (2024)

Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
by: Shao, Ruichen, et al.
Published: (2025)

DPO-Shift: Shifting the Distribution of Direct Preference Optimization
by: Yang, Xiliang, et al.
Published: (2025)

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence
by: Lu, Junru, et al.
Published: (2024)

On the Role of Preference Variance in Preference Optimization
by: Guo, Jiacheng, et al.
Published: (2025)

Filtered Direct Preference Optimization
by: Morimura, Tetsuro, et al.
Published: (2024)

Direct Preference Optimization with an Offset
by: Amini, Afra, et al.
Published: (2024)

Robust Preference Optimization via Dynamic Target Margins
by: Sun, Jie, et al.
Published: (2025)

Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty
by: Xiao, Yao, et al.
Published: (2025)

DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
by: Deng, Mengyi, et al.
Published: (2026)

Direct Multi-Turn Preference Optimization for Language Agents
by: Shi, Wentao, et al.
Published: (2024)

Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization
by: Li, Jian, et al.
Published: (2025)

Reverse Preference Optimization for Complex Instruction Following
by: Huang, Xiang, et al.
Published: (2025)

Direct Preference Knowledge Distillation for Large Language Models
by: Li, Yixing, et al.
Published: (2024)

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
by: Xiao, Teng, et al.
Published: (2024)

Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding
by: Yang, Guangyu, et al.
Published: (2023)

Statistical Rejection Sampling Improves Preference Optimization
by: Liu, Tianqi, et al.
Published: (2023)

Accelerating Direct Preference Optimization with Prefix Sharing
by: Wang, Franklin, et al.
Published: (2024)

Multiplayer Nash Preference Optimization
by: Wu, Fang, et al.
Published: (2025)

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
by: Guo, Yiju, et al.
Published: (2024)

Entropy Controllable Direct Preference Optimization
by: Omura, Motoki, et al.
Published: (2024)

Intrinsic Mutual Information as a Modulator for Preference Optimization
by: Liao, Peng, et al.
Published: (2026)

Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization
by: Li, Xintong, et al.
Published: (2025)

Style Transfer with Multi-iteration Preference Optimization
by: Liu, Shuai, et al.
Published: (2024)

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
by: Lin, Yong, et al.
Published: (2024)