Image de couverture de livre

Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yao, Chaorui, Chen, Yanxi, Sun, Yuchang, Chen, Yushuo, Zhang, Wenhao, Pan, Xuchen, Li, Yaliang, Ding, Bolin
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Artificial Intelligence Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
par: Zhang, Wenhao, et autres
Publié: (2025)

Enhancing Latent Computation in Transformers with Latent Tokens
par: Sun, Yuchang, et autres
Publié: (2025)

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
par: Chen, Yanxi, et autres
Publié: (2023)

Provable Scaling Laws for the Test-Time Compute of Large Language Models
par: Chen, Yanxi, et autres
Publié: (2024)

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
par: Pan, Xuchen, et autres
Publié: (2024)

Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
par: Chen, Yanxi, et autres
Publié: (2024)

Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models
par: Pan, Xuchen, et autres
Publié: (2025)

Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs
par: Wei, Fei, et autres
Publié: (2025)

R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification
par: Shi, Weijie, et autres
Publié: (2026)

Very Large-Scale Multi-Agent Simulation in AgentScope
par: Pan, Xuchen, et autres
Publié: (2024)

On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
par: Wang, Shumin, et autres
Publié: (2026)

Exploring Selective Layer Fine-Tuning in Federated Learning
par: Sun, Yuchang, et autres
Publié: (2024)

Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs
par: Roux, Nicolas Le, et autres
Publié: (2025)

Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
par: Zhou, Hongyi, et autres
Publié: (2026)

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
par: Tian, Minghao, et autres
Publié: (2026)

Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models
par: Chen, Daoyuan, et autres
Publié: (2024)

Towards Anthropomorphic Conversational AI Part I: A Practical Framework
par: Wei, Fei, et autres
Publié: (2025)

OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models
par: Zhang, Liyu, et autres
Publié: (2026)

LLM-Based Multi-Agent Systems are Scalable Graph Generative Models
par: Ji, Jiarui, et autres
Publié: (2024)

GenSim: A General Social Simulation Platform with Large Language Model based Agents
par: Tang, Jiakai, et autres
Publié: (2024)

Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning
par: Deng, Jingcheng, et autres
Publié: (2026)

It Takes Two: Your GRPO Is Secretly DPO
par: Wu, Yihong, et autres
Publié: (2025)

Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model
par: Zhou, Renping, et autres
Publié: (2025)

UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
par: Wang, Jiaqi, et autres
Publié: (2026)

MotionGRPO: Overcoming Low Intra-Group Diversity in GRPO-Based Egocentric Motion Recovery
par: Yao, Nanjie, et autres
Publié: (2026)

BiMix: A Bivariate Data Mixing Law for Language Model Pretraining
par: Ge, Ce, et autres
Publié: (2024)

DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization
par: Xie, Xuan, et autres
Publié: (2025)

Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization
par: Cang, Yueyang, et autres
Publié: (2026)

SeeUPO: Sequence-Level Agentic-RL with Convergence Guarantees
par: Hu, Tianyi, et autres
Publié: (2026)

AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering
par: Cai, Yuzhu, et autres
Publié: (2026)

Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
par: Zhang, Xichen, et autres
Publié: (2025)

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
par: Wang, Jingyi, et autres
Publié: (2026)

Improving LoRA in Privacy-preserving Federated Learning
par: Sun, Youbang, et autres
Publié: (2024)

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
par: Chen, Minghan, et autres
Publié: (2025)

The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
par: Qin, Zhen, et autres
Publié: (2024)

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning
par: Mundada, Gagan, et autres
Publié: (2026)

F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking
par: Surana, Rohan, et autres
Publié: (2026)

Clustering Mixtures of Discrete Distributions: A Note on Mitra's Algorithm
par: Seif, Mohamed, et autres
Publié: (2024)

Leveraging LLM-based agents for social science research: insights from citation network simulations
par: Ji, Jiarui, et autres
Publié: (2025)

QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE
par: Zhao, Junjie, et autres
Publié: (2024)