Cover Image

Mitigating the Alignment Tax of RLHF

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lin, Yong, Lin, Hangyu, Xiong, Wei, Diao, Shizhe, Liu, Jianmeng, Zhang, Jipeng, Pan, Rui, Wang, Haoxiang, Hu, Wenbin, Zhang, Hanning, Dong, Hanze, Pi, Renjie, Zhao, Han, Jiang, Nan, Ji, Heng, Yao, Yuan, Zhang, Tong
Format:	Preprint
Published:	2023
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Instinctive Bias: Spurious Images lead to Illusion in MLLMs
by: Han, Tianyang, et al.
Published: (2024)

LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
by: Pan, Rui, et al.
Published: (2024)

TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts
by: Wang, Ruida, et al.
Published: (2024)

LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models
by: Diao, Shizhe, et al.
Published: (2023)

Entropy-Regularized Process Reward Model
by: Zhang, Hanning, et al.
Published: (2024)

Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training
by: Pan, Rui, et al.
Published: (2025)

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
by: Wang, Haoxiang, et al.
Published: (2024)

RLHF Workflow: From Reward Modeling to Online RLHF
by: Dong, Hanze, et al.
Published: (2024)

MA-LoT: Model-Collaboration Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Proving
by: Wang, Ruida, et al.
Published: (2025)

Plum: Prompt Learning using Metaheuristic
by: Pan, Rui, et al.
Published: (2023)

ScaleBiO: Scalable Bilevel Optimization for LLM Data Reweighting
by: Pan, Rui, et al.
Published: (2024)

Effective Bilevel Optimization via Minimax Reformulation
by: Wang, Xiaoyu, et al.
Published: (2023)

MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance
by: Pi, Renjie, et al.
Published: (2024)

Personalized Visual Instruction Tuning
by: Pi, Renjie, et al.
Published: (2024)

Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions
by: Pi, Renjie, et al.
Published: (2024)

TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data
by: Zhang, Jipeng, et al.
Published: (2024)

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
by: Xiong, Wei, et al.
Published: (2023)

Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization
by: Pi, Renjie, et al.
Published: (2024)

R-Tuning: Instructing Large Language Models to Say `I Don't Know'
by: Zhang, Hanning, et al.
Published: (2023)

Generalizable Geometric Image Caption Synthesis
by: Xin, Yue, et al.
Published: (2025)

Active Prompting with Chain-of-Thought for Large Language Models
by: Diao, Shizhe, et al.
Published: (2023)

VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
by: Zhang, Jipeng, et al.
Published: (2025)

Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
by: Yao, Jiarui, et al.
Published: (2025)

Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code
by: Zhang, Jipeng, et al.
Published: (2024)

FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation
by: Shum, KaShun, et al.
Published: (2024)

GAR: Generative Adversarial Reinforcement Learning for Formal Theorem Proving
by: Wang, Ruida, et al.
Published: (2025)

Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data
by: Shum, KaShun, et al.
Published: (2023)

Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
by: Chen, Ziyi, et al.
Published: (2025)

Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
by: Ye, Chenlu, et al.
Published: (2024)

Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment
by: Lu, Keming, et al.
Published: (2024)

Self-rewarding correction for mathematical reasoning
by: Xiong, Wei, et al.
Published: (2025)

ROCM: RLHF on consistency models
by: Shekhar, Shivanshu, et al.
Published: (2025)

AlignGuard: Scalable Safety Alignment for Text-to-Image Generation
by: Liu, Runtao, et al.
Published: (2024)

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
by: Hao, Yifan, et al.
Published: (2025)

ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects
by: Zhang, Jipeng, et al.
Published: (2025)

Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
by: Sun, Guanglong, et al.
Published: (2026)

Reward Shaping to Mitigate Reward Hacking in RLHF
by: Fu, Jiayi, et al.
Published: (2025)

PhysProver: Advancing Automatic Theorem Proving for Physics
by: Zhang, Hanning, et al.
Published: (2026)

Policy Filtration for RLHF to Mitigate Noise in Reward Models
by: Zhang, Chuheng, et al.
Published: (2024)

Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
by: Zhu, Yu, et al.
Published: (2024)