Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.11522 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914469340250112 |
|---|---|
| author | Wang, Xinda Hou, Zhengxu Zhang, Yangshijie Yan, Bingren Liu, Jialin Zhao, Chenzhuo Yang, Zhibo Yang, Bin-Bin Xiao, Feng |
| author_facet | Wang, Xinda Hou, Zhengxu Zhang, Yangshijie Yan, Bingren Liu, Jialin Zhao, Chenzhuo Yang, Zhibo Yang, Bin-Bin Xiao, Feng |
| contents | Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_11522 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Triviality Corrected Endogenous Reward Wang, Xinda Hou, Zhengxu Zhang, Yangshijie Yan, Bingren Liu, Jialin Zhao, Chenzhuo Yang, Zhibo Yang, Bin-Bin Xiao, Feng Computation and Language Reinforcement learning for open-ended text generation is constrained by the lack of verifiable rewards, necessitating reliance on judge models that require either annotated data or powerful closed-source models. Inspired by recent work on unsupervised reinforcement learning for mathematical reasoning using confidence-based endogenous rewards, we investigate whether this principle can be adapted to open-ended writing tasks. We find that directly applying confidence rewards leads to Triviality Bias: the policy collapses toward high-probability outputs, reducing diversity and meaningful content. We propose TCER (Triviality Corrected Endogenous Reward), which addresses this bias by rewarding the relative information gain between a specialist policy and a generalist reference policy, modulated by a probability-dependent correction mechanism. Across multiple writing benchmarks and model architectures, TCER achieves consistent improvements without external supervision. Furthermore, TCER also transfers effectively to mathematical reasoning, validating the generality of our approach across different generation tasks. |
| title | Triviality Corrected Endogenous Reward |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.11522 |