Imagen de Portada

On the Role of Preference Variance in Preference Optimization

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Guo, Jiacheng, Li, Zihao, Qiu, Jiahao, Wu, Yue, Wang, Mengdi
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Artificial Intelligence Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Temporal Consistency for LLM Reasoning Process Error Identification
por: Guo, Jiacheng, et al.
Publicado: (2025)

MaxMin-RLHF: Alignment with Diverse Human Preferences
por: Chakraborty, Souradip, et al.
Publicado: (2024)

ROPO: Robust Preference Optimization for Large Language Models
por: Liang, Xize, et al.
Publicado: (2024)

Self-Play Preference Optimization for Language Model Alignment
por: Wu, Yue, et al.
Publicado: (2024)

From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models
por: Zhu, Mingkang, et al.
Publicado: (2025)

Geometric-Averaged Preference Optimization for Soft Preference Labels
por: Furuta, Hiroki, et al.
Publicado: (2024)

Orthogonal Finetuning for Direct Preference Optimization
por: Yang, Chenxu, et al.
Publicado: (2024)

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
por: Wu, Junkang, et al.
Publicado: (2024)

Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
por: Jian, Chengtao, et al.
Publicado: (2025)

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
por: Gupta, Taneesh, et al.
Publicado: (2025)

Filtered Direct Preference Optimization
por: Morimura, Tetsuro, et al.
Publicado: (2024)

Direct Preference Optimization with an Offset
por: Amini, Afra, et al.
Publicado: (2024)

Self-Consistency Preference Optimization
por: Prasad, Archiki, et al.
Publicado: (2024)

TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling
por: Qiu, Jiahao, et al.
Publicado: (2024)

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
por: Wang, Haoxiang, et al.
Publicado: (2024)

TSO: Self-Training with Scaled Preference Optimization
por: Chen, Kaihui, et al.
Publicado: (2024)

Entropy Controllable Direct Preference Optimization
por: Omura, Motoki, et al.
Publicado: (2024)

HelpSteer2-Preference: Complementing Ratings with Preferences
por: Wang, Zhilin, et al.
Publicado: (2024)

Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization
por: Li, Meng, et al.
Publicado: (2025)

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
por: Wu, Junkang, et al.
Publicado: (2024)

Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
por: Liu, Yinhong, et al.
Publicado: (2024)

Preference Optimization by Estimating the Ratio of the Data Distribution
por: Kim, Yeongmin, et al.
Publicado: (2025)

Active Preference Optimization for Sample Efficient RLHF
por: Das, Nirjhar, et al.
Publicado: (2024)

VPO: Leveraging the Number of Votes in Preference Optimization
por: Cho, Jae Hyeon, et al.
Publicado: (2024)

WPO: Enhancing RLHF with Weighted Preference Optimization
por: Zhou, Wenxuan, et al.
Publicado: (2024)

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
por: Xiao, Wenyi, et al.
Publicado: (2024)

Adaptive Margin RLHF via Preference over Preferences
por: Chittepu, Yaswanth, et al.
Publicado: (2025)

Preference Learning Algorithms Do Not Learn Preference Rankings
por: Chen, Angelica, et al.
Publicado: (2024)

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
por: Qi, Biqing, et al.
Publicado: (2024)

Preference Poisoning Attacks on Reward Model Learning
por: Wu, Junlin, et al.
Publicado: (2024)

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
por: Wang, Jialu, et al.
Publicado: (2026)

Accelerated Preference Optimization for Large Language Model Alignment
por: He, Jiafan, et al.
Publicado: (2024)

T-REG: Preference Optimization with Token-Level Reward Regularization
por: Zhou, Wenxuan, et al.
Publicado: (2024)

Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
por: Razin, Noam, et al.
Publicado: (2024)

Preference Leakage: A Contamination Problem in LLM-as-a-judge
por: Li, Dawei, et al.
Publicado: (2025)

Larger or Smaller Reward Margins to Select Preferences for Alignment?
por: Huang, Kexin, et al.
Publicado: (2025)

Quantifying and Mitigating Self-Preference Bias of LLM Judges
por: Yang, Jinming, et al.
Publicado: (2026)

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game
por: Cheng, Pengyu, et al.
Publicado: (2023)

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering
por: Mohamed, Anas, et al.
Publicado: (2025)

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
por: Zhang, Yifan, et al.
Publicado: (2024)