Towards Interpretable Counterfactual Generation via Multimodal Autoregression
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914016717176832 |
|---|---|
| author | Ma, Chenglong Ji, Yuanfeng Ye, Jin Zhang, Lu Chen, Ying Li, Tianbin Li, Mingjie He, Junjun Shan, Hongming |
| author_facet | Ma, Chenglong Ji, Yuanfeng Ye, Jin Zhang, Lu Chen, Ying Li, Tianbin Li, Mingjie He, Junjun Shan, Hongming |
| contents | Counterfactual medical image generation enables clinicians to explore clinical hypotheses, such as predicting disease progression, facilitating their decision-making. While existing methods can generate visually plausible images from disease progression prompts, they produce silent predictions that lack interpretation to verify how the generation reflects the hypothesized progression -- a critical gap for medical applications that require traceable reasoning. In this paper, we propose Interpretable Counterfactual Generation (ICG), a novel task requiring the joint generation of counterfactual images that reflect the clinical hypothesis and interpretation texts that outline the visual changes induced by the hypothesis. To enable ICG, we present ICG-CXR, the first dataset pairing longitudinal medical images with hypothetical progression prompts and textual interpretations. We further introduce ProgEmu, an autoregressive model that unifies the generation of counterfactual images and textual interpretations. We demonstrate the superiority of ProgEmu in generating progression-aligned counterfactuals and interpretations, showing significant potential in enhancing clinical decision support and medical education. Project page: https://progemu.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23149 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Interpretable Counterfactual Generation via Multimodal Autoregression Ma, Chenglong Ji, Yuanfeng Ye, Jin Zhang, Lu Chen, Ying Li, Tianbin Li, Mingjie He, Junjun Shan, Hongming Image and Video Processing Counterfactual medical image generation enables clinicians to explore clinical hypotheses, such as predicting disease progression, facilitating their decision-making. While existing methods can generate visually plausible images from disease progression prompts, they produce silent predictions that lack interpretation to verify how the generation reflects the hypothesized progression -- a critical gap for medical applications that require traceable reasoning. In this paper, we propose Interpretable Counterfactual Generation (ICG), a novel task requiring the joint generation of counterfactual images that reflect the clinical hypothesis and interpretation texts that outline the visual changes induced by the hypothesis. To enable ICG, we present ICG-CXR, the first dataset pairing longitudinal medical images with hypothetical progression prompts and textual interpretations. We further introduce ProgEmu, an autoregressive model that unifies the generation of counterfactual images and textual interpretations. We demonstrate the superiority of ProgEmu in generating progression-aligned counterfactuals and interpretations, showing significant potential in enhancing clinical decision support and medical education. Project page: https://progemu.github.io. |
| title | Towards Interpretable Counterfactual Generation via Multimodal Autoregression |
| topic | Image and Video Processing |
| url | https://arxiv.org/abs/2503.23149 |