Cover Image

Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xu, Yuanda, Sang, Hejian, Zhou, Zhengze, He, Ran, Wang, Zhipeng, Geramifard, Alborz
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

TIP: Token Importance in On-Policy Distillation
by: Xu, Yuanda, et al.
Published: (2026)

PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence
by: Xu, Yuanda, et al.
Published: (2026)

Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning
by: Xu, Yuanda, et al.
Published: (2026)

Not all tokens are needed(NAT): token efficient reinforcement learning
by: Sang, Hejian, et al.
Published: (2026)

CRISP: Compressed Reasoning via Iterative Self-Policy Distillation
by: Sang, Hejian, et al.
Published: (2026)

Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs
by: Zhu, Siyu, et al.
Published: (2025)

SODA: Semi On-Policy Black-Box Distillation for Large Language Models
by: Chen, Xiwen, et al.
Published: (2026)

Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
by: Markovic-Voronov, Jelena, et al.
Published: (2026)

When should we prefer Decision Transformers for Offline Reinforcement Learning?
by: Bhargava, Prajjwal, et al.
Published: (2023)

DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment
by: Deng, Haoyou, et al.
Published: (2026)

SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning
by: Sikchi, Harshit, et al.
Published: (2023)

Aligning Diffusion Language Models via Unpaired Preference Optimization
by: Jindal, Vaibhav, et al.
Published: (2025)

Sequential Decision-Making for Inline Text Autocomplete
by: Chitnis, Rohan, et al.
Published: (2024)

Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policies
by: Khanzada, Feeza Khan, et al.
Published: (2025)

Debunk the Myth of SFT Generalization
by: Lin, Xiaofeng, et al.
Published: (2025)

Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
by: Chen, Wei-Rui, et al.
Published: (2025)

GRPO and Reflection Reward for Mathematical Reasoning in Large Language Models
by: Wang, Zhijie
Published: (2026)

TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models
by: Ding, Zheng, et al.
Published: (2025)

MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting
by: Wei, Kangda, et al.
Published: (2026)

GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
by: Tan, Hongze, et al.
Published: (2025)

GRPO is Secretly a Process Reward Model
by: Sullivan, Michael, et al.
Published: (2025)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
by: Pan, Bowen, et al.
Published: (2024)

Efficient Paths and Dense Rewards: Probabilistic Flow Reasoning for Large Language Models
by: Liu, Yan, et al.
Published: (2026)

ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Training
by: Ai, Rui, et al.
Published: (2026)

Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
by: Afsharrad, Amirhossein, et al.
Published: (2026)

Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
by: Zhao, Shiwan, et al.
Published: (2026)

Agentic Reinforcement Learning for Real-World Code Repair
by: Zhu, Siyu, et al.
Published: (2025)

Intrinsic Reward Policy Optimization for Sparse-Reward Environments
by: Cho, Minjae, et al.
Published: (2026)

Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
by: Wu, Yecheng, et al.
Published: (2026)

Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
by: Cao, Meng, et al.
Published: (2024)

Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retention
by: Lin, Wenye, et al.
Published: (2026)

SFT-GRPO Data Overlap as a Post-Training Hyperparameter for Autoformalization
by: Su, Xiaole, et al.
Published: (2026)

SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
by: Chen, Minghan, et al.
Published: (2025)

Shaping Explanations: Semantic Reward Modeling with Encoder-Only Transformers for GRPO
by: Pappone, Francesco, et al.
Published: (2025)

Beyond Policy Optimization: A Data Curation Flywheel for Sparse-Reward Long-Horizon Planning
by: Wang, Yutong, et al.
Published: (2025)

Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
by: Zhang, Junkai, et al.
Published: (2025)

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO
by: Ren, Yiming, et al.
Published: (2026)

OISD: On-Policy Internal Self-Distillation of Language Models
by: Liu, Xinyu, et al.
Published: (2026)

HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime
by: Sana, Mohamed, et al.
Published: (2026)

RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agents
by: Zhong, Haitian, et al.
Published: (2026)