Cover Image

COPR: Continual Learning Human Preference through Optimal Policy Regularization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Han, Gui, Lin, Zhai, Yuanzhao, Wang, Hui, Lei, Yu, Xu, Ruifeng
Format:	Preprint
Published:	2023
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

COPR: Continual Human Preference Learning via Optimal Policy Regularization
by: Zhang, Han, et al.
Published: (2024)

Correcting Large Language Model Behavior via Influence Function
by: Zhang, Han, et al.
Published: (2024)

Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
by: Zhai, Yuanzhao, et al.
Published: (2023)

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
by: Wang, Tianze, et al.
Published: (2025)

Online Self-Preferring Language Models
by: Zhai, Yuanzhao, et al.
Published: (2024)

Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
by: Peng, Andi, et al.
Published: (2024)

LiPO: Listwise Preference Optimization through Learning-to-Rank
by: Liu, Tianqi, et al.
Published: (2024)

Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
by: Bai, Fengshuo, et al.
Published: (2024)

T-REG: Preference Optimization with Token-Level Reward Regularization
by: Zhou, Wenxuan, et al.
Published: (2024)

Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
by: Qiu, Wenjie, et al.
Published: (2025)

LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization
by: Lu, Peng, et al.
Published: (2023)

Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values
by: Bell, Henry, et al.
Published: (2026)

Understanding Reference Policies in Direct Preference Optimization
by: Liu, Yixin, et al.
Published: (2024)

Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Models
by: Zhang, Zhicheng, et al.
Published: (2025)

Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment
by: Li, Moxin, et al.
Published: (2025)

Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond
by: Wang, Xinyu, et al.
Published: (2024)

Length Desensitization in Direct Preference Optimization
by: Liu, Wei, et al.
Published: (2024)

Off-Policy Value-Based Reinforcement Learning for Large Language Models
by: Wang, Peng-Yuan, et al.
Published: (2026)

Optimizing Language Models for Human Preferences is a Causal Inference Problem
by: Lin, Victoria, et al.
Published: (2024)

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
by: Yin, Yueqin, et al.
Published: (2024)

Preference Learning Algorithms Do Not Learn Preference Rankings
by: Chen, Angelica, et al.
Published: (2024)

MaxMin-RLHF: Alignment with Diverse Human Preferences
by: Chakraborty, Souradip, et al.
Published: (2024)

Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization
by: Li, Meng, et al.
Published: (2025)

Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
by: Xu, Nuo, et al.
Published: (2024)

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
by: Wang, Haoxiang, et al.
Published: (2024)

Entropy-Regularized Process Reward Model
by: Zhang, Hanning, et al.
Published: (2024)

Mobility-LLM: Learning Visiting Intentions and Travel Preferences from Human Mobility Data with Large Language Models
by: Gong, Letian, et al.
Published: (2024)

Correct and Optimal: the Regular Expression Inference Challenge
by: Valizadeh, Mojtaba, et al.
Published: (2023)

Decoding the Ear: A Framework for Objectifying Expressiveness from Human Preference Through Efficient Alignment
by: Lin, Zhiyu, et al.
Published: (2025)

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
by: Wang, Haoxiang, et al.
Published: (2024)

Internalize the Temperature: On-Policy Self-Distillation as Policy Reheater for Reinforcement Learning
by: Yang, Xuewei, et al.
Published: (2026)

DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning
by: Wan, Zhongwei, et al.
Published: (2026)

Preference learning made easy: Everything should be understood through win rate
by: Zhang, Lily H., et al.
Published: (2025)

How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation
by: Lee, Wilson Y.
Published: (2026)

Improving RL Exploration for LLM Reasoning through Retrospective Replay
by: Dou, Shihan, et al.
Published: (2025)

On-Policy RL with Optimal Reward Baseline
by: Hao, Yaru, et al.
Published: (2025)

ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
by: Cai, Tianchi, et al.
Published: (2023)

Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data
by: Zhao, Shuai, et al.
Published: (2025)

$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization
by: Han, Jiaqi, et al.
Published: (2024)

Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
by: Zhai, Yuanzhao, et al.
Published: (2024)