Cover Image

Offline and Online KL-Regularized RLHF under Differential Privacy

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wu, Yulian, Thareja, Rushil, Vepakomma, Praneeth, Orabona, Francesco
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DP-Fusion: Token-Level Differentially Private Inference for Large Language Models
by: Thareja, Rushil, et al.
Published: (2025)

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
by: Zhou, Xingyu, et al.
Published: (2025)

Predicting Survival of Hemodialysis Patients using Federated Learning
by: Raju, Abhiram, et al.
Published: (2024)

KL-regularization Itself is Differentially Private in Bandits and RLHF
by: Zhang, Yizhou, et al.
Published: (2025)

MAC: Multi-Agent Constitution Learning
by: Thareja, Rushil, et al.
Published: (2026)

Square$χ$PO: Differentially Private and Robust $χ^2$-Preference Optimization in Offline Direct Alignment
by: Zhou, Xingyu, et al.
Published: (2025)

Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
by: Liu, Kezhao, et al.
Published: (2025)

Combinatorial Privacy: Private Multi-Party Bitstream Grand Sum by Hiding in Birkhoff Polytopes
by: Vepakomma, Praneeth
Published: (2026)

Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
by: Tang, Kenton, et al.
Published: (2026)

ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
by: Singhal, Raghav, et al.
Published: (2025)

Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study
by: Ponkshe, Kaustubh, et al.
Published: (2025)

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
by: Cen, Shicong, et al.
Published: (2024)

Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
by: Chen, Ziyi, et al.
Published: (2025)

Optimal Regret of Bernoulli Bandits under Global Differential Privacy
by: Azize, Achraf, et al.
Published: (2025)

Better-than-KL PAC-Bayes Bounds
by: Kuzborskij, Ilja, et al.
Published: (2024)

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
by: Xiong, Wei, et al.
Published: (2023)

On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularization
by: Ji, Kaixuan, et al.
Published: (2026)

Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
by: Zhao, Qingyue, et al.
Published: (2026)

RLHF Workflow: From Reward Modeling to Online RLHF
by: Dong, Hanze, et al.
Published: (2024)

Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
by: Ponkshe, Kaustubh, et al.
Published: (2024)

On the Exponential Convergence for Offline RLHF with Pairwise Comparisons
by: Chen, Zhirui, et al.
Published: (2024)

Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
by: Dai, Juntao, et al.
Published: (2025)

$f$-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
by: Wu, Di, et al.
Published: (2026)

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
by: Liu, Zhihan, et al.
Published: (2024)

Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning
by: Singhal, Raghav, et al.
Published: (2025)

Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
by: Gao, Chen-Xiao, et al.
Published: (2025)

Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
by: He, Longxiang, et al.
Published: (2025)

SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance
by: Zhang, Liyu, et al.
Published: (2024)

Mutual Information Regularized Offline Reinforcement Learning
by: Ma, Xiao, et al.
Published: (2022)

Federated Offline Policy Optimization with Dual Regularization
by: Yue, Sheng, et al.
Published: (2024)

Latent Adversarial Regularization for Offline Preference Optimization
by: Jiang, Enyi, et al.
Published: (2026)

Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework
by: Kim, Kihyun, et al.
Published: (2025)

A Comedy of Estimators: On KL Regularization in RL Training of LLMs
by: Shah, Vedant, et al.
Published: (2025)

Power Mechanism: Private Tabular Representation Release for Model Agnostic Consumption
by: Vepakomma, Praneeth, et al.
Published: (2025)

A Modern Introduction to Online Learning
by: Orabona, Francesco
Published: (2019)

Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning
by: Chen, Haohui, et al.
Published: (2025)

Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning
by: Ghanem, Abdelghani, et al.
Published: (2026)

The Role of Deep Learning Regularizations on Actors in Offline RL
by: Tarasov, Denis, et al.
Published: (2024)

Streamlining Admission with LOR Insights: AI-Based Leadership Assessment in Online Master's Program
by: Soylu, Meryem Yilmaz, et al.
Published: (2025)

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning
by: Bozkurt, Alper Kamil, et al.
Published: (2026)