Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918104901091328 |
|---|---|
| author | Li, Xinshu Wang, Ruoyu Gao, Erdun Gong, Mingming Yao, Lina |
| author_facet | Li, Xinshu Wang, Ruoyu Gao, Erdun Gong, Mingming Yao, Lina |
| contents | Prompt learning has garnered attention for its efficiency over traditional model training and fine-tuning. However, existing methods, constrained by inadequate theoretical foundations, encounter difficulties in achieving causally invariant prompts, ultimately falling short of capturing robust features that generalize effectively across categories. To address these challenges, we introduce the $\textit{\textbf{DiCap}}$ model, a theoretically grounded $\textbf{Di}$ffusion-based $\textbf{C}$ounterf$\textbf{a}$ctual $\textbf{p}$rompt learning framework, which leverages a diffusion process to iteratively sample gradients from the marginal and conditional distributions of the causal model, guiding the generation of counterfactuals that satisfy the minimal sufficiency criterion. Grounded in rigorous theoretical derivations, this approach guarantees the identifiability of counterfactual outcomes while imposing strict bounds on estimation errors. We further employ a contrastive learning framework that leverages the generated counterfactuals, thereby enabling the refined extraction of prompts that are precisely aligned with the causal features of the data. Extensive experimental results demonstrate that our method performs excellently across tasks such as image classification, image-text retrieval, and visual question answering, with particularly strong advantages in unseen categories. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_19882 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation Li, Xinshu Wang, Ruoyu Gao, Erdun Gong, Mingming Yao, Lina Artificial Intelligence Prompt learning has garnered attention for its efficiency over traditional model training and fine-tuning. However, existing methods, constrained by inadequate theoretical foundations, encounter difficulties in achieving causally invariant prompts, ultimately falling short of capturing robust features that generalize effectively across categories. To address these challenges, we introduce the $\textit{\textbf{DiCap}}$ model, a theoretically grounded $\textbf{Di}$ffusion-based $\textbf{C}$ounterf$\textbf{a}$ctual $\textbf{p}$rompt learning framework, which leverages a diffusion process to iteratively sample gradients from the marginal and conditional distributions of the causal model, guiding the generation of counterfactuals that satisfy the minimal sufficiency criterion. Grounded in rigorous theoretical derivations, this approach guarantees the identifiability of counterfactual outcomes while imposing strict bounds on estimation errors. We further employ a contrastive learning framework that leverages the generated counterfactuals, thereby enabling the refined extraction of prompts that are precisely aligned with the causal features of the data. Extensive experimental results demonstrate that our method performs excellently across tasks such as image classification, image-text retrieval, and visual question answering, with particularly strong advantages in unseen categories. |
| title | Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.19882 |