Cover Image

Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Xiaoying, Ton, Jean-Francois, Shen, Wei, Wang, Hongning, Liu, Yang
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer
by: Liu, Zhihan, et al.
Published: (2024)

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment
by: Shen, Yunyi, et al.
Published: (2025)

Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
by: Kim, Sunghwan, et al.
Published: (2025)

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
by: Zhu, Banghua, et al.
Published: (2024)

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
by: Rafailov, Rafael, et al.
Published: (2024)

Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
by: Sun, Hao, et al.
Published: (2025)

Trust-Region Adaptive Policy Optimization
by: Su, Mingyu, et al.
Published: (2025)

ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization
by: Zhang, Chen Bo Calvin, et al.
Published: (2024)

DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
by: Li, Gang, et al.
Published: (2025)

Pretrain Value, Not Reward: Decoupled Value Policy Optimization
by: Huang, Chenghua, et al.
Published: (2025)

Understanding Chain-of-Thought in LLMs through Information Theory
by: Ton, Jean-Francois, et al.
Published: (2024)

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
by: Huang, Audrey, et al.
Published: (2024)

Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
by: Liu, Yang, et al.
Published: (2023)

Value-Free Policy Optimization via Reward Partitioning
by: Faye, Bilal, et al.
Published: (2025)

Intrinsic Reward Policy Optimization for Sparse-Reward Environments
by: Cho, Minjae, et al.
Published: (2026)

Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
by: Nguyen, Thanh, et al.
Published: (2024)

Adversarial Training for Process Reward Models
by: Juneja, Gurusha, et al.
Published: (2025)

Policy Filtration for RLHF to Mitigate Noise in Reward Models
by: Zhang, Chuheng, et al.
Published: (2024)

Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
by: Chen, Yang, et al.
Published: (2025)

Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
by: Chen, Ziyi, et al.
Published: (2025)

RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
by: Ren, Tao, et al.
Published: (2025)

Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
by: Beigi, Mohammad, et al.
Published: (2026)

Selective Preference Optimization via Token-Level Reward Function Estimation
by: Yang, Kailai, et al.
Published: (2024)

Overcoming Overfitting in Reinforcement Learning via Gaussian Process Diffusion Policy
by: Horprasert, Amornyos, et al.
Published: (2025)

UCPO: Uncertainty-Aware Policy Optimization
by: Zeng, Xianzhou, et al.
Published: (2026)

ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
by: Patel, Nirmal, et al.
Published: (2026)

CROP: Conservative Reward for Model-based Offline Policy Optimization
by: Li, Hao, et al.
Published: (2023)

GOPO: Policy Optimization using Ranked Rewards
by: Choi, Kyuseong, et al.
Published: (2026)

Measuring and Reducing LLM Hallucination without Gold-Standard Answers
by: Wei, Jiaheng, et al.
Published: (2024)

Mutual-Taught for Co-adapting Policy and Reward Models
by: Shi, Tianyuan, et al.
Published: (2025)

Enhancing Adversarial Training via Reweighting Optimization Trajectory
by: Huang, Tianjin, et al.
Published: (2023)

Detector-Evasive LLM Paraphrasing via Constrained Policy Optimization
by: Wang, Mingyi, et al.
Published: (2026)

POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles
by: Menet, Nicolas, et al.
Published: (2026)

Hindsight-Anchored Policy Optimization: Turning Failure into Feedback in Sparse Reward Settings
by: Wu, Yuning, et al.
Published: (2026)

Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation
by: Jia, Chengxing, et al.
Published: (2024)

EvaDrive: Evolutionary Adversarial Policy Optimization for End-to-End Autonomous Driving
by: Jiao, Siwen, et al.
Published: (2025)

PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoning
by: Feng, Wenfeng, et al.
Published: (2025)

Invariant Learning via Probability of Sufficient and Necessary Causes
by: Yang, Mengyue, et al.
Published: (2023)

PerPO: Perceptual Preference Optimization via Discriminative Rewarding
by: Zhu, Zining, et al.
Published: (2025)

HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime
by: Sana, Mohamed, et al.
Published: (2026)