Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909080047583232 |
|---|---|
| author | Rose, Daniel Himakunthala, Vaishnavi Ouyang, Andy He, Ryan Mei, Alex Lu, Yujie Saxon, Michael Sonar, Chinmay Mirza, Diba Wang, William Yang |
| author_facet | Rose, Daniel Himakunthala, Vaishnavi Ouyang, Andy He, Ryan Mei, Alex Lu, Yujie Saxon, Michael Sonar, Chinmay Mirza, Diba Wang, William Yang |
| contents | Recent advances in large language models elicit reasoning in a chain-of-thought that allows models to decompose problems in a human-like fashion. Though this paradigm improves multi-step reasoning ability in language models, it is limited by being unimodal and applied mainly to question-answering tasks. We claim that incorporating visual augmentation into reasoning is essential, especially for complex, imaginative tasks. Consequently, we introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to recursively bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks that can benefit from temporal reasoning, as well as provide interpretability into models' multi-step reasoning. We apply VCoT to the Visual Storytelling and WikiHow summarization datasets and demonstrate through human evaluation that VCoT offers novel and consistent synthetic data augmentation beating chain-of-thought baselines, which can be used to enhance downstream performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_02317 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings Rose, Daniel Himakunthala, Vaishnavi Ouyang, Andy He, Ryan Mei, Alex Lu, Yujie Saxon, Michael Sonar, Chinmay Mirza, Diba Wang, William Yang Computation and Language Computer Vision and Pattern Recognition Recent advances in large language models elicit reasoning in a chain-of-thought that allows models to decompose problems in a human-like fashion. Though this paradigm improves multi-step reasoning ability in language models, it is limited by being unimodal and applied mainly to question-answering tasks. We claim that incorporating visual augmentation into reasoning is essential, especially for complex, imaginative tasks. Consequently, we introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to recursively bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks that can benefit from temporal reasoning, as well as provide interpretability into models' multi-step reasoning. We apply VCoT to the Visual Storytelling and WikiHow summarization datasets and demonstrate through human evaluation that VCoT offers novel and consistent synthetic data augmentation beating chain-of-thought baselines, which can be used to enhance downstream performance. |
| title | Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2305.02317 |