Cover Image

Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jian, Chengtao, Yang, Kai, Gao, Tianhao, Ni, Wuguang, Yang, Keying, Xiao, Bowen, Liu, Jiajun, Ouyang, Ye
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
by: Zhang, Ruiqi, et al.
Published: (2024)

FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
by: Liu, Tong, et al.
Published: (2025)

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game
by: Cheng, Pengyu, et al.
Published: (2023)

Towards Understanding the Influence of Reward Margin on Preference Model Performance
by: Qin, Bowen, et al.
Published: (2024)

LRHP: Learning Representations for Human Preferences via Preference Pairs
by: Wang, Chenglong, et al.
Published: (2024)

Preference Ranking Optimization for Human Alignment
by: Song, Feifan, et al.
Published: (2023)

Token-level Direct Preference Optimization
by: Zeng, Yongcheng, et al.
Published: (2024)

MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting
by: Li, Tianhao, et al.
Published: (2024)

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
by: Yang, Wen, et al.
Published: (2025)

Multiplayer Nash Preference Optimization
by: Wu, Fang, et al.
Published: (2025)

Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness
by: Li, Jian, et al.
Published: (2024)

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
by: Qi, Biqing, et al.
Published: (2024)

Step-level Value Preference Optimization for Mathematical Reasoning
by: Chen, Guoxin, et al.
Published: (2024)

Creative Preference Optimization
by: Ismayilzada, Mete, et al.
Published: (2025)

Enhancing Multilingual Counterfactual Generation through Alignment-as-Preference Optimization
by: Wang, Yilong, et al.
Published: (2026)

On the Role of Preference Variance in Preference Optimization
by: Guo, Jiacheng, et al.
Published: (2025)

Preference Packing: Efficient Preference Optimization for Large Language Models
by: Cho, Jaekyung
Published: (2026)

CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences
by: Pokharel, Rhitabrat, et al.
Published: (2025)

Iterative Reasoning Preference Optimization
by: Pang, Richard Yuanzhe, et al.
Published: (2024)

Weights-Rotated Preference Optimization for Large Language Models
by: Yang, Chenxu, et al.
Published: (2025)

Geometric-Averaged Preference Optimization for Soft Preference Labels
by: Furuta, Hiroki, et al.
Published: (2024)

Emotion-Aligned Generation in Diffusion Text to Speech Models via Preference-Guided Optimization
by: Shi, Jiacheng, et al.
Published: (2025)

SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder
by: Liu, Dengcan, et al.
Published: (2025)

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
by: Wu, Junkang, et al.
Published: (2024)

Orthogonal Finetuning for Direct Preference Optimization
by: Yang, Chenxu, et al.
Published: (2024)

Textual Self-attention Network: Test-Time Preference Optimization through Textual Gradient-based Attention
by: Mo, Shibing, et al.
Published: (2025)

Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level
by: Liu, Jie, et al.
Published: (2024)

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
by: Dong, Zhijin
Published: (2025)

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
by: Guo, Yiju, et al.
Published: (2024)

Full-Stack Domain Enhancement for Combustion LLMs: Construction and Optimization
by: Xiao, Quanjia, et al.
Published: (2026)

PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs
by: Liu, An, et al.
Published: (2024)

Self-Play Preference Optimization for Language Model Alignment
by: Wu, Yue, et al.
Published: (2024)

Selective Preference Optimization via Token-Level Reward Function Estimation
by: Yang, Kailai, et al.
Published: (2024)

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
by: Gupta, Taneesh, et al.
Published: (2025)

PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization
by: Wu, Jiayi, et al.
Published: (2024)

Filtered Direct Preference Optimization
by: Morimura, Tetsuro, et al.
Published: (2024)

Direct Preference Optimization with an Offset
by: Amini, Afra, et al.
Published: (2024)

Self-Consistency Preference Optimization
by: Prasad, Archiki, et al.
Published: (2024)

RPO-RAG: Aligning Small LLMs with Relation-aware Preference Optimization for Knowledge Graph Question Answering
by: Um, Kaehyun, et al.
Published: (2026)

PerPO: Perceptual Preference Optimization via Discriminative Rewarding
by: Zhu, Zining, et al.
Published: (2025)