Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910061683539968 |
|---|---|
| author | Wu, Jiayun Hou, Peixu Qu, Shan Zhang, Peng Gu, Ning Lu, Tun |
| author_facet | Wu, Jiayun Hou, Peixu Qu, Shan Zhang, Peng Gu, Ning Lu, Tun |
| contents | Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios.
We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: first-token prediction as a scalar score (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking.
F/S-RM achieves a 1.2% relative performance improvement over state-of-the-art models while reducing token consumption by 20.8%. Code and data will be publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_20212 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models Wu, Jiayun Hou, Peixu Qu, Shan Zhang, Peng Gu, Ning Lu, Tun Computation and Language Machine Learning Reward models (RMs) are critical for aligning Large Language Models via Reinforcement Learning from Human Feedback (RLHF). While Generative Reward Models (GRMs) achieve superior accuracy through chain-of-thought (CoT) reasoning, they incur substantial computational costs. Conversely, Scalar Reward Models (SRMs) offer efficiency but suffer from limited performance and adaptability in complex scenarios. We introduce Fast-Slow Thinking Reward Models (F/S-RM), a hybrid RM architecture inspired by Dual Process Theory. It trains a single model to integrate two distinct reward paradigms: first-token prediction as a scalar score (fast thinking) and CoT-based judgment (slow thinking), regulated by a dual-confidence activation mechanism that determines when to activate slow thinking. F/S-RM achieves a 1.2% relative performance improvement over state-of-the-art models while reducing token consumption by 20.8%. Code and data will be publicly available. |
| title | Fast-Slow Thinking RM: Efficient Integration of Scalar and Generative Reward Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2603.20212 |