Cover Image

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gao, Zhaolin, Zhan, Wenhao, Chang, Jonathan D., Swamy, Gokul, Brantley, Kianté, Lee, Jason D., Sun, Wen
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Dataset Reset Policy Optimization for RLHF
by: Chang, Jonathan D., et al.
Published: (2024)

REBEL: Reinforcement Learning via Regressing Relative Rewards
by: Gao, Zhaolin, et al.
Published: (2024)

Reviewer2: Optimizing Review Generation Through Prompt Generation
by: Gao, Zhaolin, et al.
Published: (2024)

Accelerating RL for LLM Reasoning with Optimal Advantage Regression
by: Brantley, Kianté, et al.
Published: (2025)

$p1$: Better Prompt Optimization with Fewer Prompts
by: Gao, Zhaolin, et al.
Published: (2026)

Value-Guided Search for Efficient Chain-of-Thought Reasoning
by: Wang, Kaiwen, et al.
Published: (2025)

Policy-Gradient Training of Language Models for Ranking
by: Gao, Ge, et al.
Published: (2023)

Diffusing States and Matching Scores: A New Framework for Imitation Learning
by: Wu, Runzhe, et al.
Published: (2024)

$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training
by: Zhou, Jin Peng, et al.
Published: (2025)

A Surprising Failure? Multimodal LLMs and the NLVR Challenge
by: Wu, Anne, et al.
Published: (2024)

Breadcrumbs Reasoning: Memory-Efficient Reasoning with Compression Beacons
by: Monea, Giovanni, et al.
Published: (2025)

Scaling Offline RL via Efficient and Expressive Shortcut Models
by: Espinosa-Dice, Nicolas, et al.
Published: (2025)

LLMs Are In-Context Bandit Reinforcement Learners
by: Monea, Giovanni, et al.
Published: (2024)

LLMs Can Learn to Reason Via Off-Policy RL
by: Ritter, Daniel, et al.
Published: (2026)

Adversarial Imitation Learning via Boosting
by: Chang, Jonathan D., et al.
Published: (2024)

RL for Consistency Models: Faster Reward Guided Text-to-Image Generation
by: Oertell, Owen, et al.
Published: (2024)

Expressive Value Learning for Scalable Offline Reinforcement Learning
by: Espinosa-Dice, Nicolas, et al.
Published: (2025)

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage
by: Song, Yuda, et al.
Published: (2024)

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
by: Huang, Audrey, et al.
Published: (2024)

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
by: Du, Yihan, et al.
Published: (2024)

Active Preference Optimization for Sample Efficient RLHF
by: Das, Nirjhar, et al.
Published: (2024)

Psy-Insight: Explainable Multi-turn Bilingual Dataset for Mental Health Counseling
by: Chen, Keqi, et al.
Published: (2025)

Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning
by: Sutawika, Lintang, et al.
Published: (2026)

Quantile Regression for Distributional Reward Models in RLHF
by: Dorka, Nicolai
Published: (2024)

AM$^3$Safety: Towards Data Efficient Alignment of Multi-modal Multi-turn Safety for MLLMs
by: Zhu, Han, et al.
Published: (2026)

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
by: Noukhovitch, Michael, et al.
Published: (2024)

Efficient Imitation under Misspecification
by: Espinosa-Dice, Nicolas, et al.
Published: (2025)

Measuring Sycophancy of Language Models in Multi-turn Dialogues
by: Hong, Jiseung, et al.
Published: (2025)

Optimizing Anytime Reasoning via Budget Relative Policy Optimization
by: Qi, Penghui, et al.
Published: (2025)

Prompt Curriculum Learning for Efficient LLM Post-Training
by: Gao, Zhaolin, et al.
Published: (2025)

Ranking with Long-Term Constraints
by: Brantley, Kianté, et al.
Published: (2023)

Solving the Inverse Alignment Problem for Efficient RLHF
by: Krishna, Shambhavi, et al.
Published: (2024)

Prototypical Reward Network for Data-Efficient RLHF
by: Zhang, Jinghan, et al.
Published: (2024)

Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing
by: Wen, Junyi, et al.
Published: (2025)

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF
by: Xie, Tengyang, et al.
Published: (2024)

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
by: Hu, Jian, et al.
Published: (2024)

MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation
by: Yang, Chenghao, et al.
Published: (2025)

Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
by: Gao, Bin, et al.
Published: (2024)

Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management
by: Lu, Miao, et al.
Published: (2025)

The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
by: Wang, Xiaobo, et al.
Published: (2026)