IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913787730198528 |
|---|---|
| author | You, Haoxuan Wang, Zhecan Sun, Rui Chen, Long Wang, Gengyu Ayyubi, Hammad A. Chang, Kai-Wei Chang, Shih-Fu |
| author_facet | You, Haoxuan Wang, Zhecan Sun, Rui Chen, Long Wang, Gengyu Ayyubi, Hammad A. Chang, Kai-Wei Chang, Shih-Fu |
| contents | The field of vision-and-language (VL) understanding has made unprecedented progress with end-to-end large pre-trained VL models (VLMs). However, they still fall short in zero-shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely on domain-specific sub-question decomposing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, a framework that iteratively decomposes VL reasoning using large language models (LLMs). Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer. These three modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. We evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE. Code is available at https://github.com/Hxyou/IdealGPT |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_14985 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models You, Haoxuan Wang, Zhecan Sun, Rui Chen, Long Wang, Gengyu Ayyubi, Hammad A. Chang, Kai-Wei Chang, Shih-Fu Computer Vision and Pattern Recognition Computation and Language The field of vision-and-language (VL) understanding has made unprecedented progress with end-to-end large pre-trained VL models (VLMs). However, they still fall short in zero-shot reasoning tasks that require multi-step inferencing. To achieve this goal, previous works resort to a divide-and-conquer pipeline. In this paper, we argue that previous efforts have several inherent shortcomings: 1) They rely on domain-specific sub-question decomposing models. 2) They force models to predict the final answer even if the sub-questions or sub-answers provide insufficient information. We address these limitations via IdealGPT, a framework that iteratively decomposes VL reasoning using large language models (LLMs). Specifically, IdealGPT utilizes an LLM to generate sub-questions, a VLM to provide corresponding sub-answers, and another LLM to reason to achieve the final answer. These three modules perform the divide-and-conquer procedure iteratively until the model is confident about the final answer to the main question. We evaluate IdealGPT on multiple challenging VL reasoning tasks under a zero-shot setting. In particular, our IdealGPT outperforms the best existing GPT-4-like models by an absolute 10% on VCR and 15% on SNLI-VE. Code is available at https://github.com/Hxyou/IdealGPT |
| title | IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2305.14985 |