Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.01258 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909156733091840 |
|---|---|
| author | Zhang, Ruohong Gui, Liangke Sun, Zhiqing Feng, Yihao Xu, Keyang Zhang, Yuanhan Fu, Di Li, Chunyuan Hauptmann, Alexander Bisk, Yonatan Yang, Yiming |
| author_facet | Zhang, Ruohong Gui, Liangke Sun, Zhiqing Feng, Yihao Xu, Keyang Zhang, Yuanhan Fu, Di Li, Chunyuan Hauptmann, Alexander Bisk, Yonatan Yang, Yiming |
| contents | Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge. Previous studies have explored using large large multimodal models (LMMs) as reward models to guide preference modeling, but their ability to accurately assess the factuality of generated responses compared to corresponding videos has not been conclusively established. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model's reward mechanism, which directly takes video frames as input. Furthermore, we show that applying this tailored reward through DPO significantly improves the performance of video LMMs on video QA tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01258 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward Zhang, Ruohong Gui, Liangke Sun, Zhiqing Feng, Yihao Xu, Keyang Zhang, Yuanhan Fu, Di Li, Chunyuan Hauptmann, Alexander Bisk, Yonatan Yang, Yiming Computer Vision and Pattern Recognition Artificial Intelligence Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for detecting hallucinations in generated responses, remains a significant challenge. Previous studies have explored using large large multimodal models (LMMs) as reward models to guide preference modeling, but their ability to accurately assess the factuality of generated responses compared to corresponding videos has not been conclusively established. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model's reward mechanism, which directly takes video frames as input. Furthermore, we show that applying this tailored reward through DPO significantly improves the performance of video LMMs on video QA tasks. |
| title | Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2404.01258 |