Cover Image

Statistical Rejection Sampling Improves Preference Optimization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Tianqi, Zhao, Yao, Joshi, Rishabh, Khalman, Misha, Saleh, Mohammad, Liu, Peter J., Liu, Jialu
Format:	Preprint
Published:	2023
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

LiPO: Listwise Preference Optimization through Learning-to-Rank
by: Liu, Tianqi, et al.
Published: (2024)

Direct Language Model Alignment from Online AI Feedback
by: Guo, Shangmin, et al.
Published: (2024)

Predicting Text Preference Via Structured Comparative Reasoning
by: Yan, Jing Nathan, et al.
Published: (2023)

PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairs
by: Zhang, Rongzhi, et al.
Published: (2024)

RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
by: Khaki, Saeed, et al.
Published: (2024)

Building Math Agents with Multi-Turn Iterative Preference Learning
by: Xiong, Wei, et al.
Published: (2024)

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play
by: Ye, Ziyu, et al.
Published: (2024)

Multilingual Fine-Grained News Headline Hallucination Detection
by: Shen, Jiaming, et al.
Published: (2024)

BPO: Revisiting Preference Modeling in Direct Preference Optimization
by: Sun, Lin, et al.
Published: (2025)

Improving Factual Consistency of News Summarization by Contrastive Preference Optimization
by: Feng, Huawen, et al.
Published: (2023)

PersoDPO: Scalable Preference Optimization for Instruction-Adherent, Persona-Grounded Dialogue via Multi-LLM Evaluation
by: Afzoon, Saleh, et al.
Published: (2026)

RRM: Robust Reward Model Training Mitigates Reward Hacking
by: Liu, Tianqi, et al.
Published: (2024)

Style Transfer with Multi-iteration Preference Optimization
by: Liu, Shuai, et al.
Published: (2024)

Preference Optimization with Multi-Sample Comparisons
by: Wang, Chaoqi, et al.
Published: (2024)

Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
by: Yao, Jiarui, et al.
Published: (2025)

Flipping Against All Odds: Reducing LLM Coin Flip Bias via Verbalized Rejection Sampling
by: Xiao, Tim Z., et al.
Published: (2025)

Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
by: Zhang, Xuan, et al.
Published: (2024)

PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation
by: Liu, Yuxuan
Published: (2025)

Understanding Reference Policies in Direct Preference Optimization
by: Liu, Yixin, et al.
Published: (2024)

Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
by: Cho, Jay Hyeon, et al.
Published: (2025)

Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs
by: Shen, Yifan, et al.
Published: (2025)

Constrained Adaptive Rejection Sampling
by: Parys, Paweł, et al.
Published: (2025)

Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models
by: Zhu, Zhaowei, et al.
Published: (2023)

Beyond Rejection Sampling: Trajectory Fusion for Scaling Mathematical Reasoning
by: Deng, Jie, et al.
Published: (2026)

TO-GATE: Clarifying Questions and Summarizing Responses with Trajectory Optimization for Eliciting Human Preference
by: Dou, Yulin, et al.
Published: (2025)

No Preference Left Behind: Group Distributional Preference Optimization
by: Yao, Binwei, et al.
Published: (2024)

AIPO: Improving Training Objective for Iterative Preference Optimization
by: Shen, Yaojie, et al.
Published: (2024)

Aligning Teacher with Student Preferences for Tailored Training Data Generation
by: Liu, Yantao, et al.
Published: (2024)

LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification
by: Qin, Zhen, et al.
Published: (2024)

Beyond Single-Reward: Multi-Pair, Multi-Perspective Preference Optimization for Machine Translation
by: Wang, Hao, et al.
Published: (2025)

RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation
by: Yan, Shi-Qi, et al.
Published: (2025)

FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
by: Liu, Tong, et al.
Published: (2025)

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence
by: Lu, Junru, et al.
Published: (2024)

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
by: Chen, Zhuotong, et al.
Published: (2024)

WEPO: Web Element Preference Optimization for LLM-based Web Navigation
by: Liu, Jiarun, et al.
Published: (2024)

Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization
by: Duan, Shaohua, et al.
Published: (2025)

Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation
by: Shen, Jiaming, et al.
Published: (2024)

TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
by: Liu, Aiwei, et al.
Published: (2024)

Toolken+: Improving LLM Tool Usage with Reranking and a Reject Option
by: Yakovlev, Konstantin, et al.
Published: (2024)

A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
by: Xiong, Wei, et al.
Published: (2025)