Cover Image

On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Yifan, Liu, Yifeng, Yuan, Huizhuo, Yuan, Yang, Gu, Quanquan, Yao, Andrew Chi-Chih
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Tensor Product Attention Is All You Need
by: Zhang, Yifan, et al.
Published: (2025)

Group Representational Position Encoding
by: Zhang, Yifan, et al.
Published: (2025)

Accelerated Preference Optimization for Large Language Model Alignment
by: He, Jiafan, et al.
Published: (2024)

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
by: Chen, Zixiang, et al.
Published: (2024)

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation
by: Yuan, Huizhuo, et al.
Published: (2024)

Self-Play Preference Optimization for Language Model Alignment
by: Wu, Yue, et al.
Published: (2024)

On the Diagram of Thought
by: Zhang, Yifan, et al.
Published: (2024)

RSPO: Regularized Self-Play Alignment of Large Language Models
by: Tang, Xiaohang, et al.
Published: (2025)

Deep Delta Learning
by: Zhang, Yifan, et al.
Published: (2026)

Meta Prompting for AI Systems
by: Zhang, Yifan, et al.
Published: (2023)

Augmenting Math Word Problems via Iterative Question Composing
by: Liu, Haoxiong, et al.
Published: (2024)

Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time
by: Chen, Zixiang, et al.
Published: (2023)

Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts
by: Zhang, Yifan, et al.
Published: (2024)

Causal Attention with Lookahead Keys
by: Song, Zhuoqing, et al.
Published: (2025)

Higher-order Linear Attention
by: Zhang, Yifan, et al.
Published: (2025)

Monadic Context Engineering
by: Zhang, Yifan, et al.
Published: (2025)

Interpreting and Controlling LLM Reasoning through Integrated Policy Gradient
by: Li, Changming, et al.
Published: (2026)

Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
by: Zhao, Qingyue, et al.
Published: (2026)

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
by: Zhang, Yifan, et al.
Published: (2024)

Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
by: Tao, Yuanzhe, et al.
Published: (2024)

ConfRover: Simultaneous Modeling of Protein Conformation and Dynamics via Autoregression
by: Shen, Yuning, et al.
Published: (2025)

Near-Optimal Regret for KL-Regularized Multi-Armed Bandits
by: Ji, Kaixuan, et al.
Published: (2026)

Beyond KL Divergence: Policy Optimization with Flexible Bregman Divergences for LLM Reasoning
by: Yuan, Rui, et al.
Published: (2026)

Probing the Lack of Stable Internal Beliefs in LLMs
by: Luo, Yifan, et al.
Published: (2026)

Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
by: Fan, Zhiyuan, et al.
Published: (2025)

Cumulative Reasoning with Large Language Models
by: Zhang, Yifan, et al.
Published: (2023)

On the Optimal Sample Complexity of Offline Multi-Armed Bandits with KL Regularization
by: Ji, Kaixuan, et al.
Published: (2026)

KL for a KL: On-Policy Distillation with Control Variate Baseline
by: Oh, Minjae, et al.
Published: (2026)

Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL
by: Yao, Jiarui, et al.
Published: (2025)

Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization
by: Yao, Jiashu, et al.
Published: (2026)

Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
by: Su, Zhenpeng, et al.
Published: (2025)

Reinforce LLM Reasoning through Multi-Agent Reflection
by: Yuan, Yurun, et al.
Published: (2025)

Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
by: Deng, Yihe, et al.
Published: (2023)

Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees
by: Wu, Yongtao, et al.
Published: (2025)

KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation
by: Shi, Jiajun, et al.
Published: (2025)

Reasoning with Sampling: Cutting at Decision Points
by: Zhou, Felix, et al.
Published: (2026)

A Systematic Survey on Large Language Models for Algorithm Design
by: Liu, Fei, et al.
Published: (2024)

Simple Policy Gradients for Reasoning with Diffusion Language Models
by: Zhan, Anthony
Published: (2025)

Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization
by: Ji, Kaixuan, et al.
Published: (2024)

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement
by: Yuan, Hui, et al.
Published: (2024)