Cover Image

OPPO: Accelerating PPO-based RLHF via Pipeline Overlap

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yan, Kaizhuo, Yu, Yingjie, Yu, Yifan, Zheng, Haizhong, Lai, Fan
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
by: Bamba, Udbhav, et al.
Published: (2025)

OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
by: Li, Yu, et al.
Published: (2026)

QoS-QoE Translation with Large Language Model
by: Yu, Yingjie, et al.
Published: (2026)

DPO Meets PPO: Reinforced Token Optimization for RLHF
by: Zhong, Han, et al.
Published: (2024)

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
by: Huang, Shengyi, et al.
Published: (2024)

What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
by: Yuan, Yufeng, et al.
Published: (2025)

Reward-Robust RLHF in LLMs
by: Yan, Yuzi, et al.
Published: (2024)

SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
by: Yin, Haofei, et al.
Published: (2025)

PPO guided Agentic Pipeline for Adaptive Prompt Selection and Test Case Generation
by: Koushik, Gourisetty Venkata Sai, et al.
Published: (2026)

Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
by: Zheng, Haizhong, et al.
Published: (2025)

Policy Optimization in RLHF: The Impact of Out-of-preference Data
by: Li, Ziniu, et al.
Published: (2023)

VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
by: Wu, Mingyuan, et al.
Published: (2025)

Accelerating Decentralized Optimization via Overlapping Local Steps
by: Zhou, Yijie, et al.
Published: (2026)

Accelerating RLHF Training with Reward Variance Increase
by: Yang, Zonglin, et al.
Published: (2025)

Directional-Clamp PPO
by: Karpel, Gilad, et al.
Published: (2025)

A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
by: Xu, Wenyuan, et al.
Published: (2025)

Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
by: Fang, Zhiyuan, et al.
Published: (2025)

Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?
by: Wu, Mingyuan, et al.
Published: (2025)

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training
by: Xiao, Youshao, et al.
Published: (2023)

Class-Proportional Coreset Selection for Difficulty-Separable Data
by: Tsai, Elisa, et al.
Published: (2025)

Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
by: Zheng, Haizhong, et al.
Published: (2025)

Provably Efficient Online RLHF with One-Pass Reward Modeling
by: Li, Long-Fei, et al.
Published: (2025)

MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding
by: Huang, Runxi, et al.
Published: (2025)

Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping
by: Jiang, Chenyu, et al.
Published: (2024)

Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
by: Dai, Juntao, et al.
Published: (2025)

Securing Transformer-based AI Execution via Unified TEEs and Crypto-protected Accelerators
by: Xue, Jiaqi, et al.
Published: (2025)

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences
by: Mukherjee, Arpan, et al.
Published: (2025)

Boosting Deductive Reasoning with Step Signals In RLHF
by: Li, Jialian, et al.
Published: (2024)

Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
by: Li, Junbo, et al.
Published: (2025)

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
by: Hu, Jian, et al.
Published: (2024)

Online Bandit Learning with Offline Preference Data for Improved RLHF
by: Agnihotri, Akhil, et al.
Published: (2024)

Sampling Complexity of TD and PPO in RKHS
by: Zou, Lu, et al.
Published: (2025)

SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
by: Chegini, Atoosa, et al.
Published: (2024)

LLM Misalignment via Adversarial RLHF Platforms
by: Entezami, Erfan, et al.
Published: (2025)

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
by: Beukman, Michael, et al.
Published: (2026)

Group Robust Preference Optimization in Reward-free RLHF
by: Ramesh, Shyam Sundhar, et al.
Published: (2024)

Efficient Federated RLHF via Zeroth-Order Policy Optimization
by: Wang, Deyi, et al.
Published: (2026)

Mitigating the Alignment Tax of RLHF
by: Lin, Yong, et al.
Published: (2023)

Factored Causal Representation Learning for Robust Reward Modeling in RLHF
by: Yang, Yupei, et al.
Published: (2026)

Learn To be Efficient: Build Structured Sparsity in Large Language Models
by: Zheng, Haizhong, et al.
Published: (2024)