Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.01483 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909981101522944 |
|---|---|
| author | Qiu, Xinyu Jia, Heng Zeng, Zhengwen Shen, Shuheng Meng, Changhua Yang, Yi Zhu, Linchao |
| author_facet | Qiu, Xinyu Jia, Heng Zeng, Zhengwen Shen, Shuheng Meng, Changhua Yang, Yi Zhu, Linchao |
| contents | Parallel test-time scaling typically trains separate generation and verification models, incurring high training and inference costs. We propose Advantage Decoupled Preference Optimization (ADPO), a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. ADPO introduces two innovations: a preference verification reward improving verification capability and a decoupled optimization mechanism enabling synergistic optimization of generation and verification. Specifically, the preference verification reward computes mean verification scores from positive and negative samples as decision thresholds, providing positive feedback when prediction correctness aligns with answer correctness. Meanwhile, the advantage decoupled optimization computes separate advantages for generation and verification, applies token masks to isolate gradients, and combines masked GRPO objectives, preserving generation quality while calibrating verification scores. ADPO achieves up to +34.1% higher verification AUC and -53.5% lower inference time, with significant gains of +2.8%/+1.4% accuracy on MathVista/MMMU, +1.9 cIoU on ReasonSeg, and +1.7%/+1.0% step success rate on AndroidControl/GUI Odyssey. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_01483 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization Qiu, Xinyu Jia, Heng Zeng, Zhengwen Shen, Shuheng Meng, Changhua Yang, Yi Zhu, Linchao Computer Vision and Pattern Recognition Parallel test-time scaling typically trains separate generation and verification models, incurring high training and inference costs. We propose Advantage Decoupled Preference Optimization (ADPO), a unified reinforcement learning framework that jointly learns answer generation and self-verification within a single policy. ADPO introduces two innovations: a preference verification reward improving verification capability and a decoupled optimization mechanism enabling synergistic optimization of generation and verification. Specifically, the preference verification reward computes mean verification scores from positive and negative samples as decision thresholds, providing positive feedback when prediction correctness aligns with answer correctness. Meanwhile, the advantage decoupled optimization computes separate advantages for generation and verification, applies token masks to isolate gradients, and combines masked GRPO objectives, preserving generation quality while calibrating verification scores. ADPO achieves up to +34.1% higher verification AUC and -53.5% lower inference time, with significant gains of +2.8%/+1.4% accuracy on MathVista/MMMU, +1.9 cIoU on ReasonSeg, and +1.7%/+1.0% step success rate on AndroidControl/GUI Odyssey. |
| title | Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.01483 |