Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.08763 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918291041157120 |
|---|---|
| author | Hu, Zhiyuan Wang, Yucheng He, Yufei Wu, Jiaying Zhao, Yilun Ng, See-Kiong Breazeal, Cynthia Luu, Anh Tuan Park, Hae Won Hooi, Bryan |
| author_facet | Hu, Zhiyuan Wang, Yucheng He, Yufei Wu, Jiaying Zhao, Yilun Ng, See-Kiong Breazeal, Cynthia Luu, Anh Tuan Park, Hae Won Hooi, Bryan |
| contents | Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_08763 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs Hu, Zhiyuan Wang, Yucheng He, Yufei Wu, Jiaying Zhao, Yilun Ng, See-Kiong Breazeal, Cynthia Luu, Anh Tuan Park, Hae Won Hooi, Bryan Machine Learning Computation and Language Reinforcement learning (RL) has become a central paradigm for post-training large language models (LLMs), particularly for complex reasoning tasks, yet it often suffers from exploration collapse: policies prematurely concentrate on a small set of dominant reasoning patterns, improving pass@1 while limiting rollout-level diversity and gains in pass@k. We argue that this failure stems from regularizing local token behavior rather than diversity over sets of solutions. To address this, we propose Uniqueness-Aware Reinforcement Learning, a rollout-level objective that explicitly rewards correct solutions that exhibit rare high-level strategies. Our method uses an LLM-based judge to cluster rollouts for the same problem according to their high-level solution strategies, ignoring superficial variations, and reweights policy advantages inversely with cluster size. As a result, correct but novel strategies receive higher rewards than redundant ones. Across mathematics, physics, and medical reasoning benchmarks, our approach consistently improves pass@$k$ across large sampling budgets and increases the area under the pass@$k$ curve (AUC@$K$) without sacrificing pass@1, while sustaining exploration and uncovering more diverse solution strategies at scale. |
| title | Rewarding the Rare: Uniqueness-Aware RL for Creative Problem Solving in LLMs |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.08763 |