Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2405.17220 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918176469549056 |
|---|---|
| author | Yu, Tianyu Zhang, Haoye Li, Qiming Xu, Qixin Yao, Yuan Chen, Da Lu, Xiaoman Cui, Ganqu Dang, Yunkai He, Taiwen Feng, Xiaocheng Song, Jun Zheng, Bo Liu, Zhiyuan Chua, Tat-Seng Sun, Maosong |
| author_facet | Yu, Tianyu Zhang, Haoye Li, Qiming Xu, Qixin Yao, Yuan Chen, Da Lu, Xiaoman Cui, Ganqu Dang, Yunkai He, Taiwen Feng, Xiaocheng Song, Jun Zheng, Bo Liu, Zhiyuan Chua, Tat-Seng Sun, Maosong |
| contents | Traditional feedback learning for hallucination reduction relies on labor-intensive manual labeling or expensive proprietary models. This leaves the community without foundational knowledge about how to build high-quality feedback with open-source MLLMs. In this work, we introduce RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm. RLAIF-V maximally explores open-source MLLMs from two perspectives, including high-quality feedback data generation for preference learning and self-feedback guidance for inference-time scaling. Extensive experiments on six benchmarks in both automatic and human evaluation show that RLAIF-V substantially enhances the trustworthiness of models at both preference learning and inference time. RLAIF-V 7B reduces object hallucination by 80.7\% and overall hallucination by 33.7\%. Remarkably, RLAIF-V 12B further reveals the self-alignment potential of open-source MLLMs, where the model can learn from feedback of itself to achieve super GPT-4V trustworthiness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17220 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness Yu, Tianyu Zhang, Haoye Li, Qiming Xu, Qixin Yao, Yuan Chen, Da Lu, Xiaoman Cui, Ganqu Dang, Yunkai He, Taiwen Feng, Xiaocheng Song, Jun Zheng, Bo Liu, Zhiyuan Chua, Tat-Seng Sun, Maosong Computation and Language Traditional feedback learning for hallucination reduction relies on labor-intensive manual labeling or expensive proprietary models. This leaves the community without foundational knowledge about how to build high-quality feedback with open-source MLLMs. In this work, we introduce RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm. RLAIF-V maximally explores open-source MLLMs from two perspectives, including high-quality feedback data generation for preference learning and self-feedback guidance for inference-time scaling. Extensive experiments on six benchmarks in both automatic and human evaluation show that RLAIF-V substantially enhances the trustworthiness of models at both preference learning and inference time. RLAIF-V 7B reduces object hallucination by 80.7\% and overall hallucination by 33.7\%. Remarkably, RLAIF-V 12B further reveals the self-alignment potential of open-source MLLMs, where the model can learn from feedback of itself to achieve super GPT-4V trustworthiness. |
| title | RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.17220 |