Copertina

Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Yao, Chaorui, Chen, Yanxi, Sun, Yuchang, Chen, Yushuo, Zhang, Wenhao, Pan, Xuchen, Li, Yaliang, Ding, Bolin
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
di: Zhang, Wenhao, et al.
Pubblicazione: (2025)

Enhancing Latent Computation in Transformers with Latent Tokens
di: Sun, Yuchang, et al.
Pubblicazione: (2025)

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
di: Chen, Yanxi, et al.
Pubblicazione: (2023)

Provable Scaling Laws for the Test-Time Compute of Large Language Models
di: Chen, Yanxi, et al.
Pubblicazione: (2024)

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
di: Pan, Xuchen, et al.
Pubblicazione: (2024)

Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
di: Chen, Yanxi, et al.
Pubblicazione: (2024)

Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models
di: Pan, Xuchen, et al.
Pubblicazione: (2025)

Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs
di: Wei, Fei, et al.
Pubblicazione: (2025)

R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification
di: Shi, Weijie, et al.
Pubblicazione: (2026)

Very Large-Scale Multi-Agent Simulation in AgentScope
di: Pan, Xuchen, et al.
Pubblicazione: (2024)

On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models
di: Wang, Shumin, et al.
Pubblicazione: (2026)

Exploring Selective Layer Fine-Tuning in Federated Learning
di: Sun, Yuchang, et al.
Pubblicazione: (2024)

Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs
di: Roux, Nicolas Le, et al.
Pubblicazione: (2025)

Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
di: Zhou, Hongyi, et al.
Pubblicazione: (2026)

How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
di: Tian, Minghao, et al.
Pubblicazione: (2026)

Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models
di: Chen, Daoyuan, et al.
Pubblicazione: (2024)

Towards Anthropomorphic Conversational AI Part I: A Practical Framework
di: Wei, Fei, et al.
Pubblicazione: (2025)

OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models
di: Zhang, Liyu, et al.
Pubblicazione: (2026)

LLM-Based Multi-Agent Systems are Scalable Graph Generative Models
di: Ji, Jiarui, et al.
Pubblicazione: (2024)

GenSim: A General Social Simulation Platform with Large Language Model based Agents
di: Tang, Jiakai, et al.
Pubblicazione: (2024)

Latent-GRPO: Group Relative Policy Optimization for Latent Reasoning
di: Deng, Jingcheng, et al.
Pubblicazione: (2026)

It Takes Two: Your GRPO Is Secretly DPO
di: Wu, Yihong, et al.
Pubblicazione: (2025)

Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model
di: Zhou, Renping, et al.
Pubblicazione: (2025)

UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
di: Wang, Jiaqi, et al.
Pubblicazione: (2026)

MotionGRPO: Overcoming Low Intra-Group Diversity in GRPO-Based Egocentric Motion Recovery
di: Yao, Nanjie, et al.
Pubblicazione: (2026)

BiMix: A Bivariate Data Mixing Law for Language Model Pretraining
di: Ge, Ce, et al.
Pubblicazione: (2024)

DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization
di: Xie, Xuan, et al.
Pubblicazione: (2025)

Graph-GRPO: Stabilizing Multi-Agent Topology Learning via Group Relative Policy Optimization
di: Cang, Yueyang, et al.
Pubblicazione: (2026)

SeeUPO: Sequence-Level Agentic-RL with Convergence Guarantees
di: Hu, Tianyi, et al.
Pubblicazione: (2026)

AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering
di: Cai, Yuzhu, et al.
Pubblicazione: (2026)

Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
di: Zhang, Xichen, et al.
Pubblicazione: (2025)

GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
di: Wang, Jingyi, et al.
Pubblicazione: (2026)

Improving LoRA in Privacy-preserving Federated Learning
di: Sun, Youbang, et al.
Pubblicazione: (2024)

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
di: Chen, Minghan, et al.
Pubblicazione: (2025)

The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective
di: Qin, Zhen, et al.
Pubblicazione: (2024)

WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoning
di: Mundada, Gagan, et al.
Pubblicazione: (2026)

F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking
di: Surana, Rohan, et al.
Pubblicazione: (2026)

Clustering Mixtures of Discrete Distributions: A Note on Mitra's Algorithm
di: Seif, Mohamed, et al.
Pubblicazione: (2024)

Leveraging LLM-based agents for social science research: insights from citation network simulations
di: Ji, Jiarui, et al.
Pubblicazione: (2025)

QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE
di: Zhao, Junjie, et al.
Pubblicazione: (2024)