Cover Image

Reward Model Overoptimisation in Iterated RLHF

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wolf, Lorenz, Kirk, Robert, Musolesi, Mirco
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Heterogeneous Knowledge for Augmented Modular Reinforcement Learning
by: Wolf, Lorenz, et al.
Published: (2023)

Emergent Semantic Role Understanding in Language Models
by: Griffiths, Carla, et al.
Published: (2026)

DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generation
by: Franceschelli, Giorgio, et al.
Published: (2025)

Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generation
by: Franceschelli, Giorgio, et al.
Published: (2025)

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
by: Zhu, Banghua, et al.
Published: (2024)

Creative Beam Search: LLM-as-a-Judge For Improving Response Generation
by: Franceschelli, Giorgio, et al.
Published: (2024)

RLHF Workflow: From Reward Modeling to Online RLHF
by: Dong, Hanze, et al.
Published: (2024)

How to Evaluate Reward Models for RLHF
by: Frick, Evan, et al.
Published: (2024)

Quantile Regression for Distributional Reward Models in RLHF
by: Dorka, Nicolai
Published: (2024)

Reward Shaping to Mitigate Reward Hacking in RLHF
by: Fu, Jiayi, et al.
Published: (2025)

Reward-Robust RLHF in LLMs
by: Yan, Yuzi, et al.
Published: (2024)

Scalable Ensembling For Mitigating Reward Overoptimisation
by: Ahmed, Ahmed M., et al.
Published: (2024)

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF
by: Lu, Taiming, et al.
Published: (2024)

Understanding the Effects of RLHF on LLM Generalisation and Diversity
by: Kirk, Robert, et al.
Published: (2023)

Information-Theoretic Reward Decomposition for Generalizable RLHF
by: Mao, Liyuan, et al.
Published: (2025)

ODIN: Disentangled Reward Mitigates Hacking in RLHF
by: Chen, Lichang, et al.
Published: (2024)

Opponent Shaping in LLM Agents
by: Segura, Marta Emili Garcia, et al.
Published: (2025)

Mitigating Reward Hacking in RLHF via Advantage Sign Robustness
by: Ono, Shinnosuke, et al.
Published: (2026)

CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks
by: Wang, Hao, et al.
Published: (2026)

Copyright in Generative Deep Learning
by: Franceschelli, Giorgio, et al.
Published: (2021)

Creativity and Machine Learning: A Survey
by: Franceschelli, Giorgio, et al.
Published: (2021)

DeepCreativity: Measuring Creativity with Deep Learning Techniques
by: Franceschelli, Giorgio, et al.
Published: (2022)

RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
by: Du, Yuhao, et al.
Published: (2025)

Reward Generalization in RLHF: A Topological Perspective
by: Qiu, Tianyi, et al.
Published: (2024)

Training Foundation Models as Data Compression: On Information, Model Weights and Copyright Law
by: Franceschelli, Giorgio, et al.
Published: (2024)

On the Creativity of Large Language Models
by: Franceschelli, Giorgio, et al.
Published: (2023)

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
by: Hu, Jian, et al.
Published: (2024)

Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
by: Franceschelli, Giorgio, et al.
Published: (2023)

Do Agents Dream of Electric Sheep?: Improving Generalization in Reinforcement Learning through Generative Learning
by: Franceschelli, Giorgio, et al.
Published: (2024)

Moral Alignment for LLM Agents
by: Tennant, Elizaveta, et al.
Published: (2024)

A Theoretical Framework for Partially Observed Reward-States in RLHF
by: Kausik, Chinmaya, et al.
Published: (2024)

RLHF and IIA: Perverse Incentives
by: Xu, Wanqiao, et al.
Published: (2023)

(Ir)rationality and Cognitive Biases in Large Language Models
by: Macmillan-Scott, Olivia, et al.
Published: (2024)

Dataset Reset Policy Optimization for RLHF
by: Chang, Jonathan D., et al.
Published: (2024)

Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
by: Zhu, Yu, et al.
Published: (2024)

RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation
by: Liang, Kaiqu, et al.
Published: (2025)

General Exploratory Bonus for Optimistic Exploration in RLHF
by: Li, Wendi, et al.
Published: (2025)

Active Preference Optimization for Sample Efficient RLHF
by: Das, Nirjhar, et al.
Published: (2024)

The Perfect Blend: Redefining RLHF with Mixture of Judges
by: Xu, Tengyu, et al.
Published: (2024)

WPO: Enhancing RLHF with Weighted Preference Optimization
by: Zhou, Wenxuan, et al.
Published: (2024)