Image de couverture de livre

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Huang, Shengyi, Noukhovitch, Michael, Hosseini, Arian, Rasul, Kashif, Wang, Weixun, Tunstall, Lewis
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
par: Noukhovitch, Michael, et autres
Publié: (2024)

OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
par: Yan, Kaizhuo, et autres
Publié: (2025)

Forecasting with Hyper-Trees
par: März, Alexander, et autres
Publié: (2024)

DPO Meets PPO: Reinforced Token Optimization for RLHF
par: Zhong, Han, et autres
Publié: (2024)

Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization
par: Nath, Swaroop, et autres
Publié: (2024)

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
par: Hu, Jian, et autres
Publié: (2024)

Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
par: Lavoie, Samuel, et autres
Publié: (2025)

Deep Reinforcement Learning with Enhanced PPO for Safe Mobile Robot Navigation
par: Taheri, Hamid, et autres
Publié: (2024)

SHARP: A Self-Evolving Human-Auditable Rubric Policy for Financial Trading Agents
par: Chen, Xiwen, et autres
Publié: (2026)

Learning Multi-Agent Communication with Contrastive Learning
par: Lo, Yat Long, et autres
Publié: (2023)

Gradient Regularization Prevents Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
par: Ackermann, Johannes, et autres
Publié: (2026)

Recurrent Interpolants for Probabilistic Time Series Prediction
par: Chen, Yu, et autres
Publié: (2024)

ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India
par: Nigam, Shubham Kumar, et autres
Publié: (2025)

Directional-Clamp PPO
par: Karpel, Gilad, et autres
Publié: (2025)

Learning Robust Social Strategies with Large Language Models
par: Piche, Dereck, et autres
Publié: (2025)

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences
par: Mukherjee, Arpan, et autres
Publié: (2025)

Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning
par: Sinha, Sanchit, et autres
Publié: (2025)

Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models
par: Roger, Alexis, et autres
Publié: (2025)

Explaining and Preventing Alignment Collapse in Iterative RLHF
par: Gauthier, Etienne, et autres
Publié: (2026)

Optimal Design for Reward Modeling in RLHF
par: Scheid, Antoine, et autres
Publié: (2024)

Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
par: Rojas, Kevin, et autres
Publié: (2025)

Structural Knowledge Informed Continual Multivariate Time Series Forecasting
par: Pan, Zijie, et autres
Publié: (2024)

Not All LLM Reasoners Are Created Equal
par: Hosseini, Arian, et autres
Publié: (2024)

Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
par: Li, Junbo, et autres
Publié: (2025)

Mitigating the Alignment Tax of RLHF
par: Lin, Yong, et autres
Publié: (2023)

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
par: Beukman, Michael, et autres
Publié: (2026)

Sampling Complexity of TD and PPO in RKHS
par: Zou, Lu, et autres
Publié: (2025)

RLHF Workflow: From Reward Modeling to Online RLHF
par: Dong, Hanze, et autres
Publié: (2024)

PPO guided Agentic Pipeline for Adaptive Prompt Selection and Test Case Generation
par: Koushik, Gourisetty Venkata Sai, et autres
Publié: (2026)

Towards a Theoretical Understanding to the Generalization of RLHF
par: Li, Zhaochun, et autres
Publié: (2026)

PPO in the Fisher-Rao geometry
par: Lascu, Razvan-Andrei, et autres
Publié: (2025)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
par: Qiu, Zihan, et autres
Publié: (2025)

Unifying Stable Optimization and Reference Regularization in RLHF
par: He, Li, et autres
Publié: (2026)

Enhancing PPO with Trajectory-Aware Hybrid Policies
par: Liu, Qisai, et autres
Publié: (2025)

Proximal Point Nash Learning from Human Feedback
par: Tiapkin, Daniil, et autres
Publié: (2025)

Generative Verifiers: Reward Modeling as Next-Token Prediction
par: Zhang, Lunjun, et autres
Publié: (2024)

Enhancing RLHF with Human Gaze Modeling
par: Galliamov, Karim, et autres
Publié: (2025)

A Comparative Study of Deep Reinforcement Learning Models: DQN vs PPO vs A2C
par: De La Fuente, Neil, et autres
Publié: (2024)

Factored Causal Representation Learning for Robust Reward Modeling in RLHF
par: Yang, Yupei, et autres
Publié: (2026)

Beyond RLHF: A Unified Theoretical Framework of Alignment
par: Yun, Jihun, et autres
Publié: (2025)