Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915507169394688 |
|---|---|
| author | Gu, Jihao Wang, Yingyao Cao, Meng Bu, Pi Song, Jun He, Yancheng Li, Shilong Zheng, Bo |
| author_facet | Gu, Jihao Wang, Yingyao Cao, Meng Bu, Pi Song, Jun He, Yancheng Li, Shilong Zheng, Bo |
| contents | Direct Preference Optimization (DPO) has been demonstrated to be highly effective in mitigating hallucinations in Large Vision Language Models (LVLMs) by aligning their outputs more closely with human preferences. Despite the recent progress, existing methods suffer from two drawbacks: 1) Lack of scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this end, we propose a novel Token Preference Optimization model with self-calibrated rewards (dubbed as TPO), which adaptively attends to visual-correlated tokens without fine-grained annotations. Specifically, we introduce a token-level \emph{visual-anchored} \emph{reward} as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one. In addition, to highlight the informative visual-anchored tokens, a visual-aware training objective is proposed to enhance more accurate token-level optimization. Extensive experimental results have manifested the state-of-the-art performance of the proposed TPO. For example, by building on top of LLAVA-1.5-7B, our TPO boosts the performance absolute improvement for hallucination benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14487 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation Gu, Jihao Wang, Yingyao Cao, Meng Bu, Pi Song, Jun He, Yancheng Li, Shilong Zheng, Bo Computer Vision and Pattern Recognition Direct Preference Optimization (DPO) has been demonstrated to be highly effective in mitigating hallucinations in Large Vision Language Models (LVLMs) by aligning their outputs more closely with human preferences. Despite the recent progress, existing methods suffer from two drawbacks: 1) Lack of scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this end, we propose a novel Token Preference Optimization model with self-calibrated rewards (dubbed as TPO), which adaptively attends to visual-correlated tokens without fine-grained annotations. Specifically, we introduce a token-level \emph{visual-anchored} \emph{reward} as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one. In addition, to highlight the informative visual-anchored tokens, a visual-aware training objective is proposed to enhance more accurate token-level optimization. Extensive experimental results have manifested the state-of-the-art performance of the proposed TPO. For example, by building on top of LLAVA-1.5-7B, our TPO boosts the performance absolute improvement for hallucination benchmarks. |
| title | Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.14487 |