Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908610586476544 |
|---|---|
| author | Cheng, Zihui Chen, Qiguang Xu, Xiao Wang, Jiaqi Wang, Weiyun Fei, Hao Wang, Yidong Wang, Alex Jinpeng Chen, Zhi Che, Wanxiang Qin, Libo |
| author_facet | Cheng, Zihui Chen, Qiguang Xu, Xiao Wang, Jiaqi Wang, Weiyun Fei, Hao Wang, Yidong Wang, Alex Jinpeng Chen, Zhi Che, Wanxiang Qin, Libo |
| contents | Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood. To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating visual thoughts, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ in clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers, enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15510 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought Cheng, Zihui Chen, Qiguang Xu, Xiao Wang, Jiaqi Wang, Weiyun Fei, Hao Wang, Yidong Wang, Alex Jinpeng Chen, Zhi Che, Wanxiang Qin, Libo Computer Vision and Pattern Recognition Computation and Language Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood. To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating visual thoughts, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ in clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers, enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research. |
| title | Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2505.15510 |