Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.19100 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918034255380480 |
|---|---|
| author | Wang, Yeyuan Gao, Dehong Long, Rujiao Yi, Lei Jin, Linbo Yang, Libin Cai, Xiaoyan |
| author_facet | Wang, Yeyuan Gao, Dehong Long, Rujiao Yi, Lei Jin, Linbo Yang, Libin Cai, Xiaoyan |
| contents | Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19100 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning Wang, Yeyuan Gao, Dehong Long, Rujiao Yi, Lei Jin, Linbo Yang, Libin Cai, Xiaoyan Computation and Language Computer Vision and Pattern Recognition Direct Preference Optimization (DPO) has gained significant attention for its simplicity and computational efficiency in aligning large language models (LLMs). Recent advancements have extended DPO to multimodal scenarios, achieving strong performance. However, traditional DPO relies on binary preference optimization, rewarding or penalizing entire responses without considering fine-grained segment correctness, leading to suboptimal solutions. The root of this issue lies in the absence of fine-grained supervision during the optimization process. To address this, we propose Adaptive Sentence-level Preference Optimization (ASPO), which evaluates individual sentences for more precise preference optimization. By dynamically calculating adaptive rewards at the sentence level based on model predictions, ASPO enhances response content assessment without additional models or parameters. This significantly improves the alignment of multimodal features. Extensive experiments show that ASPO substantially enhances the overall performance of multimodal models. |
| title | ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.19100 |