Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911416262328320 |
|---|---|
| author | Yang, Shidong Huang, Tongwen Wen, Hao Wang, Yong Chen, Li Chu, Xiangxiang |
| author_facet | Yang, Shidong Huang, Tongwen Wen, Hao Wang, Yong Chen, Li Chu, Xiangxiang |
| contents | Multimodal reward models are crucial for aligning multimodal large language models with human preferences. Recent works have incorporated reasoning capabilities into these models, achieving promising results. However, training these models suffers from two critical challenges: (1) the inherent noise in preference datasets, which degrades model performance, and (2) the inefficiency of conventional training methods, which ignore the differences in sample difficulty. In this paper, we identify a strong correlation between response entropy and accuracy, indicating that entropy can serve as a reliable and unsupervised proxy for annotation noise and sample difficulty. Based on this insight, we propose a novel Entropy-Guided Training (EGT) approach for multimodal reasoning reward models, which combines two strategies: (1) entropy-guided data curation to mitigate the impact of unreliable samples, and (2) an entropy-guided training strategy that progressively introduces more complex examples. Extensive experiments across three benchmarks show that the EGT-trained model consistently outperforms state-of-the-art multimodal reward models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01884 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models Yang, Shidong Huang, Tongwen Wen, Hao Wang, Yong Chen, Li Chu, Xiangxiang Artificial Intelligence Machine Learning Multimodal reward models are crucial for aligning multimodal large language models with human preferences. Recent works have incorporated reasoning capabilities into these models, achieving promising results. However, training these models suffers from two critical challenges: (1) the inherent noise in preference datasets, which degrades model performance, and (2) the inefficiency of conventional training methods, which ignore the differences in sample difficulty. In this paper, we identify a strong correlation between response entropy and accuracy, indicating that entropy can serve as a reliable and unsupervised proxy for annotation noise and sample difficulty. Based on this insight, we propose a novel Entropy-Guided Training (EGT) approach for multimodal reasoning reward models, which combines two strategies: (1) entropy-guided data curation to mitigate the impact of unreliable samples, and (2) an entropy-guided training strategy that progressively introduces more complex examples. Extensive experiments across three benchmarks show that the EGT-trained model consistently outperforms state-of-the-art multimodal reward models. |
| title | Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Models |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2602.01884 |