Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.01754 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916657728847872 |
|---|---|
| author | Wu, Guande Song, Huan Wang, Yawei Yan, Qiaojing Tian, Yijun Cheong, Lin Lee Xu, Panpan |
| author_facet | Wu, Guande Song, Huan Wang, Yawei Yan, Qiaojing Tian, Yijun Cheong, Lin Lee Xu, Panpan |
| contents | Reasoning is increasingly crucial for various tasks. While chain-of-thought prompting enables large language models to leverage reasoning effectively, harnessing the reasoning capabilities of Vision-Language Models (VLMs) remains challenging. To solve this problem, we propose a novel self-distillation framework that enhances the reasoning capabilities of the model. The proposed framework introduces several key innovations. We start by employing a prompt library tailored to visual reasoning tasks to generate diverse in-context questions and utilize a two-step reasoning procedure to derive reasoning-guided responses. These responses are then used for self-distillation, enabling the model to internalize the reasoning process. Additionally, we improve the model architecture with several innovative components, including an intervention adapter for efficient parameter updates, a cross-modal skip connection to facilitate information exchange between modalities, and an ensemble learning algorithm to integrate diverse reasoning from multiple in-context questions. Extensive experiments show that our method significantly improves the baseline performance across five VQA datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01754 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Traces Wu, Guande Song, Huan Wang, Yawei Yan, Qiaojing Tian, Yijun Cheong, Lin Lee Xu, Panpan Computer Vision and Pattern Recognition Reasoning is increasingly crucial for various tasks. While chain-of-thought prompting enables large language models to leverage reasoning effectively, harnessing the reasoning capabilities of Vision-Language Models (VLMs) remains challenging. To solve this problem, we propose a novel self-distillation framework that enhances the reasoning capabilities of the model. The proposed framework introduces several key innovations. We start by employing a prompt library tailored to visual reasoning tasks to generate diverse in-context questions and utilize a two-step reasoning procedure to derive reasoning-guided responses. These responses are then used for self-distillation, enabling the model to internalize the reasoning process. Additionally, we improve the model architecture with several innovative components, including an intervention adapter for efficient parameter updates, a cross-modal skip connection to facilitate information exchange between modalities, and an ensemble learning algorithm to integrate diverse reasoning from multiple in-context questions. Extensive experiments show that our method significantly improves the baseline performance across five VQA datasets. |
| title | SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Traces |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.01754 |