Cover Image

Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shan, Zhao, Fan, Chenyou, Qiu, Shuang, Shi, Jiyuan, Bai, Chenjia
Format:	Preprint
Published:	2024
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner
by: Fan, Chenyou, et al.
Published: (2024)

Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
by: Yu, Xudong, et al.
Published: (2024)

Online Preference Alignment for Language Models via Count-based Exploration
by: Bai, Chenjia, et al.
Published: (2025)

Radiology Report Generation via Multi-objective Preference Optimization
by: Xiao, Ting, et al.
Published: (2024)

Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
by: Qiao, Zhongjian, et al.
Published: (2025)

Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
by: Zhao, Qingyue, et al.
Published: (2026)

Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning
by: Qiu, Shuang, et al.
Published: (2022)

Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
by: Shu, Yao, et al.
Published: (2026)

Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasets
by: Qiao, Zhongjian, et al.
Published: (2026)

Humanoid Whole-Body Locomotion on Narrow Terrain via Dynamic Balance and Reinforcement Learning
by: Xie, Weiji, et al.
Published: (2025)

Aligning Diffusion Language Models via Unpaired Preference Optimization
by: Jindal, Vaibhav, et al.
Published: (2025)

SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models
by: Yang, Xiaomeng, et al.
Published: (2025)

Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
by: He, Haoran, et al.
Published: (2024)

Score Regularized Policy Optimization through Diffusion Behavior
by: Chen, Huayu, et al.
Published: (2023)

Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective
by: Zhang, Yang, et al.
Published: (2025)

VLP: Vision-Language Preference Learning for Embodied Manipulation
by: Liu, Runze, et al.
Published: (2025)

Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
by: Zhang, Tianle, et al.
Published: (2024)

Preference-Based Self-Distillation: Beyond KL Matching via Reward Regularization
by: Yu, Xin, et al.
Published: (2026)

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
by: Huang, Audrey, et al.
Published: (2024)

Sharp Analysis for KL-Regularized Contextual Bandits and RLHF
by: Zhao, Heyang, et al.
Published: (2024)

Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization
by: Gu, Yi, et al.
Published: (2024)

Logarithmic Regret for Online KL-Regularized Reinforcement Learning
by: Zhao, Heyang, et al.
Published: (2025)

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
by: Zhou, Zhenglin, et al.
Published: (2025)

Cross-Domain Policy Adaptation by Capturing Representation Mismatch
by: Lyu, Jiafei, et al.
Published: (2024)

KL Penalty Control via Perturbation for Direct Preference Optimization
by: Lee, Sangkyu, et al.
Published: (2025)

On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning
by: Zhang, Yifan, et al.
Published: (2025)

Bradley-Terry Policy Optimization for Generative Preference Modeling
by: Feng, Shengyu, et al.
Published: (2025)

Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
by: Gao, Chen-Xiao, et al.
Published: (2025)

Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser
by: Ou, Zijing, et al.
Published: (2026)

Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL
by: Yao, Wei, et al.
Published: (2025)

Aligning CodeLLMs with Direct Preference Optimization
by: Miao, Yibo, et al.
Published: (2024)

Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization
by: Xing, Shuo, et al.
Published: (2025)

Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
by: Liu, Kezhao, et al.
Published: (2025)

Align-DA: Align Score-based Atmospheric Data Assimilation with Multiple Preferences
by: Sun, Jing-An, et al.
Published: (2025)

FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
by: Ma, Chiyu, et al.
Published: (2026)

Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
by: Zhu, Zhaoyu, et al.
Published: (2026)

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
by: Xiao, Jiancong, et al.
Published: (2024)

T-REG: Preference Optimization with Token-Level Reward Regularization
by: Zhou, Wenxuan, et al.
Published: (2024)

Latent Adversarial Regularization for Offline Preference Optimization
by: Jiang, Enyi, et al.
Published: (2026)

STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning
by: Luan, Yao, et al.
Published: (2025)