Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915784151793664 |
|---|---|
| author | Yang, Yankai Long, Yancheng Wei, Hongyang Chen, Wei Zhang, Tianke Jiang, Kaiyu Fan, Haonan Liu, Changyi Chen, Jiankang Tang, Kaiyu Wen, Bin Yang, Fan Gao, Tingting Li, Han Yang, Shuo |
| author_facet | Yang, Yankai Long, Yancheng Wei, Hongyang Chen, Wei Zhang, Tianke Jiang, Kaiyu Fan, Haonan Liu, Changyi Chen, Jiankang Tang, Kaiyu Wen, Bin Yang, Fan Gao, Tingting Li, Han Yang, Shuo |
| contents | Reward models are critical for reinforcement learning from human feedback, as they determine the alignment quality and reliability of generative models. For complex tasks such as image editing, reward models are required to capture global semantic consistency and implicit logical constraints beyond local similarity. Existing reward modeling approaches have clear limitations. Discriminative reward models align well with human preferences but struggle with complex semantics due to limited reasoning supervision. Generative reward models offer stronger semantic understanding and reasoning, but they are costly at inference time and difficult to align directly with human preferences. To this end, we propose Joint Reward Modeling (JRM), which jointly optimizes preference learning and language modeling on a shared vision-language backbone. This approach internalizes the semantic and reasoning capabilities of generative models into efficient discriminative representations, enabling fast and accurate evaluation. JRM achieves state-of-the-art results on MMRB2 and EditReward-Bench, and significantly improves stability and performance in downstream online reinforcement learning. These results show that joint training effectively bridges efficiency and semantic understanding in reward modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07533 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models Yang, Yankai Long, Yancheng Wei, Hongyang Chen, Wei Zhang, Tianke Jiang, Kaiyu Fan, Haonan Liu, Changyi Chen, Jiankang Tang, Kaiyu Wen, Bin Yang, Fan Gao, Tingting Li, Han Yang, Shuo Artificial Intelligence Reward models are critical for reinforcement learning from human feedback, as they determine the alignment quality and reliability of generative models. For complex tasks such as image editing, reward models are required to capture global semantic consistency and implicit logical constraints beyond local similarity. Existing reward modeling approaches have clear limitations. Discriminative reward models align well with human preferences but struggle with complex semantics due to limited reasoning supervision. Generative reward models offer stronger semantic understanding and reasoning, but they are costly at inference time and difficult to align directly with human preferences. To this end, we propose Joint Reward Modeling (JRM), which jointly optimizes preference learning and language modeling on a shared vision-language backbone. This approach internalizes the semantic and reasoning capabilities of generative models into efficient discriminative representations, enabling fast and accurate evaluation. JRM achieves state-of-the-art results on MMRB2 and EditReward-Bench, and significantly improves stability and performance in downstream online reinforcement learning. These results show that joint training effectively bridges efficiency and semantic understanding in reward modeling. |
| title | Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.07533 |