Imagen de Portada

OPPO: Accelerating PPO-based RLHF via Pipeline Overlap

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Yan, Kaizhuo, Yu, Yingjie, Yu, Yifan, Zheng, Haizhong, Lai, Fan
Formato:	Preprint
Publicado:	2025
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation
por: Bamba, Udbhav, et al.
Publicado: (2025)

OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning
por: Li, Yu, et al.
Publicado: (2026)

QoS-QoE Translation with Large Language Model
por: Yu, Yingjie, et al.
Publicado: (2026)

DPO Meets PPO: Reinforced Token Optimization for RLHF
por: Zhong, Han, et al.
Publicado: (2024)

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
por: Huang, Shengyi, et al.
Publicado: (2024)

What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret
por: Yuan, Yufeng, et al.
Publicado: (2025)

Reward-Robust RLHF in LLMs
por: Yan, Yuzi, et al.
Publicado: (2024)

SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
por: Yin, Haofei, et al.
Publicado: (2025)

PPO guided Agentic Pipeline for Adaptive Prompt Selection and Test Case Generation
por: Koushik, Gourisetty Venkata Sai, et al.
Publicado: (2026)

Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
por: Zheng, Haizhong, et al.
Publicado: (2025)

Policy Optimization in RLHF: The Impact of Out-of-preference Data
por: Li, Ziniu, et al.
Publicado: (2023)

VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
por: Wu, Mingyuan, et al.
Publicado: (2025)

Accelerating Decentralized Optimization via Overlapping Local Steps
por: Zhou, Yijie, et al.
Publicado: (2026)

Accelerating RLHF Training with Reward Variance Increase
por: Yang, Zonglin, et al.
Publicado: (2025)

Directional-Clamp PPO
por: Karpel, Gilad, et al.
Publicado: (2025)

A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
por: Xu, Wenyuan, et al.
Publicado: (2025)

Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
por: Fang, Zhiyuan, et al.
Publicado: (2025)

Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?
por: Wu, Mingyuan, et al.
Publicado: (2025)

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training
por: Xiao, Youshao, et al.
Publicado: (2023)

Class-Proportional Coreset Selection for Difficulty-Separable Data
por: Tsai, Elisa, et al.
Publicado: (2025)

Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
por: Zheng, Haizhong, et al.
Publicado: (2025)

Provably Efficient Online RLHF with One-Pass Reward Modeling
por: Li, Long-Fei, et al.
Publicado: (2025)

MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding
por: Huang, Runxi, et al.
Publicado: (2025)

Lancet: Accelerating Mixture-of-Experts Training via Whole Graph Computation-Communication Overlapping
por: Jiang, Chenyu, et al.
Publicado: (2024)

Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
por: Dai, Juntao, et al.
Publicado: (2025)

Securing Transformer-based AI Execution via Unified TEEs and Crypto-protected Accelerators
por: Xue, Jiaqi, et al.
Publicado: (2025)

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences
por: Mukherjee, Arpan, et al.
Publicado: (2025)

Boosting Deductive Reasoning with Step Signals In RLHF
por: Li, Jialian, et al.
Publicado: (2024)

Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
por: Li, Junbo, et al.
Publicado: (2025)

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
por: Hu, Jian, et al.
Publicado: (2024)

Online Bandit Learning with Offline Preference Data for Improved RLHF
por: Agnihotri, Akhil, et al.
Publicado: (2024)

Sampling Complexity of TD and PPO in RKHS
por: Zou, Lu, et al.
Publicado: (2025)

SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF
por: Chegini, Atoosa, et al.
Publicado: (2024)

LLM Misalignment via Adversarial RLHF Platforms
por: Entezami, Erfan, et al.
Publicado: (2025)

Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
por: Beukman, Michael, et al.
Publicado: (2026)

Group Robust Preference Optimization in Reward-free RLHF
por: Ramesh, Shyam Sundhar, et al.
Publicado: (2024)

Efficient Federated RLHF via Zeroth-Order Policy Optimization
por: Wang, Deyi, et al.
Publicado: (2026)

Mitigating the Alignment Tax of RLHF
por: Lin, Yong, et al.
Publicado: (2023)

Factored Causal Representation Learning for Robust Reward Modeling in RLHF
por: Yang, Yupei, et al.
Publicado: (2026)

Learn To be Efficient: Build Structured Sparsity in Large Language Models
por: Zheng, Haizhong, et al.
Publicado: (2024)