Cover Image

Reward Model Ensembles Help Mitigate Overoptimization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Coste, Thomas, Anwar, Usman, Kirk, Robert, Krueger, David
Format:	Preprint
Published:	2023
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
by: Zhu, Banghua, et al.
Published: (2024)

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
by: Eisenstein, Jacob, et al.
Published: (2023)

Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
by: Kim, Sunghwan, et al.
Published: (2025)

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
by: Rafailov, Rafael, et al.
Published: (2024)

Interpreting Emergent Planning in Model-Free Reinforcement Learning
by: Bush, Thomas, et al.
Published: (2025)

Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases
by: Zhang, Ziyi, et al.
Published: (2024)

Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation
by: Zhang, Xiaoying, et al.
Published: (2024)

Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games
by: Anwar, Usman, et al.
Published: (2024)

Comparing Bottom-Up and Top-Down Steering Approaches on In-Context Learning Tasks
by: Brumley, Madeline, et al.
Published: (2024)

Mitigating Goal Misgeneralization via Minimax Regret
by: Sadek, Karim Abdel, et al.
Published: (2025)

Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens
by: Anwar, Usman, et al.
Published: (2024)

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
by: Liu, Zhihan, et al.
Published: (2024)

Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
by: Chen, Ziyi, et al.
Published: (2025)

Scalable Ensembling For Mitigating Reward Overoptimisation
by: Ahmed, Ahmed M., et al.
Published: (2024)

Bayesian Reward Models for LLM Alignment
by: Yang, Adam X., et al.
Published: (2024)

Reward Model Overoptimisation in Iterated RLHF
by: Wolf, Lorenz, et al.
Published: (2025)

Learning to Forget using Hypernetworks
by: Rangel, Jose Miguel Lara, et al.
Published: (2024)

Defining and Characterizing Reward Hacking
by: Skalse, Joar, et al.
Published: (2022)

Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking
by: Miao, Yuchun, et al.
Published: (2025)

Debate Helps Weak Judges Reward Stronger Models
by: Elasky, Ethan, et al.
Published: (2026)

Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks
by: Jain, Samyak, et al.
Published: (2023)

Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble
by: Luo, Fan-Ming, et al.
Published: (2022)

Reward Hacking Mitigation using Verifiable Composite Rewards
by: Tarek, Mirza Farhan Bin, et al.
Published: (2025)

Repairing Reward Functions with Feedback to Mitigate Reward Hacking
by: Hatgis-Kessell, Stephane, et al.
Published: (2025)

Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders
by: Liu, Shunchang, et al.
Published: (2026)

Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning
by: Mathew, Sheryl, et al.
Published: (2025)

UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking
by: Fu, Lingling, et al.
Published: (2025)

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
by: Miao, Yuchun, et al.
Published: (2024)

Policy Filtration for RLHF to Mitigate Noise in Reward Models
by: Zhang, Chuheng, et al.
Published: (2024)

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
by: Huang, Audrey, et al.
Published: (2024)

The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards
by: Huang, Sukai, et al.
Published: (2024)

The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
by: Fluri, Lukas, et al.
Published: (2024)

SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models
by: Lian, Jiesong, et al.
Published: (2025)

A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
by: Li, Mengqi, et al.
Published: (2025)

Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
by: Beigi, Mohammad, et al.
Published: (2026)

Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction
by: Song, Ruike, et al.
Published: (2025)

Reward Shaping to Mitigate Reward Hacking in RLHF
by: Fu, Jiayi, et al.
Published: (2025)

Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
by: Zhao, Kangwen, et al.
Published: (2025)

Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
by: Liu, Zixuan, et al.
Published: (2026)

Probabilistic Modelling is Sufficient for Causal Inference
by: Mlodozeniec, Bruno, et al.
Published: (2025)