A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915373547257856 |
|---|---|
| author | Jia, Zexi Huang, Chuanwei Fei, Hongyan Zhu, Yeshuang Yuan, Zhiqiang Deng, Ying Zhang, Jiapei Zhang, Jinchao Zhou, Jie |
| author_facet | Jia, Zexi Huang, Chuanwei Fei, Hongyan Zhu, Yeshuang Yuan, Zhiqiang Deng, Ying Zhang, Jiapei Zhang, Jinchao Zhou, Jie |
| contents | Vision-language models (VLMs) often struggle with compositional reasoning due to insufficient high-quality image-text data. To tackle this challenge, we propose a novel block-based diffusion approach that automatically generates counterfactual datasets without manual annotation. Our method utilizes large language models to identify entities and their spatial relationships. It then independently generates image blocks as "puzzle pieces" coherently arranged according to specified compositional rules. This process creates diverse, high-fidelity counterfactual image-text pairs with precisely controlled variations. In addition, we introduce a specialized loss function that differentiates inter-set from intra-set samples, enhancing training efficiency and reducing the need for negative samples. Experiments demonstrate that fine-tuning VLMs with our counterfactual datasets significantly improves visual reasoning performance. Our approach achieves state-of-the-art results across multiple benchmarks while using substantially less training data than existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04699 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets Jia, Zexi Huang, Chuanwei Fei, Hongyan Zhu, Yeshuang Yuan, Zhiqiang Deng, Ying Zhang, Jiapei Zhang, Jinchao Zhou, Jie Computer Vision and Pattern Recognition Vision-language models (VLMs) often struggle with compositional reasoning due to insufficient high-quality image-text data. To tackle this challenge, we propose a novel block-based diffusion approach that automatically generates counterfactual datasets without manual annotation. Our method utilizes large language models to identify entities and their spatial relationships. It then independently generates image blocks as "puzzle pieces" coherently arranged according to specified compositional rules. This process creates diverse, high-fidelity counterfactual image-text pairs with precisely controlled variations. In addition, we introduce a specialized loss function that differentiates inter-set from intra-set samples, enhancing training efficiency and reducing the need for negative samples. Experiments demonstrate that fine-tuning VLMs with our counterfactual datasets significantly improves visual reasoning performance. Our approach achieves state-of-the-art results across multiple benchmarks while using substantially less training data than existing methods. |
| title | A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.04699 |