Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917382720585728 |
|---|---|
| author | Xu, Tianze Zheng, Yanzhao Lu, Pengrui Ye, Lyumanshan Wu, Yong Zhang, Zhentao Yu, Yuanqiang Ma, Chao Zhu, Jihuai Liu, Pengfei Dong, Baohua Zhu, Hangcheng Huang, Ruohui Yu, Gang |
| author_facet | Xu, Tianze Zheng, Yanzhao Lu, Pengrui Ye, Lyumanshan Wu, Yong Zhang, Zhentao Yu, Yuanqiang Ma, Chao Zhu, Jihuai Liu, Pengfei Dong, Baohua Zhu, Hangcheng Huang, Ruohui Yu, Gang |
| contents | Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_02795 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks Xu, Tianze Zheng, Yanzhao Lu, Pengrui Ye, Lyumanshan Wu, Yong Zhang, Zhentao Yu, Yuanqiang Ma, Chao Zhu, Jihuai Liu, Pengfei Dong, Baohua Zhu, Hangcheng Huang, Ruohui Yu, Gang Computation and Language Artificial Intelligence Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models. |
| title | Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.02795 |