Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912961184923648 |
|---|---|
| author | Huang, Xin Liang, Junjie Hou, Qingshan Cao, Peng Yang, Jinzhu Liu, Xiaoli Zaiane, Osmar R. |
| author_facet | Huang, Xin Liang, Junjie Hou, Qingshan Cao, Peng Yang, Jinzhu Liu, Xiaoli Zaiane, Osmar R. |
| contents | Medical image synthesis is crucial for alleviating data scarcity and privacy constraints. However, fine-tuning general text-to-image (T2I) models remains challenging, mainly due to the significant modality gap between complex visual details and abstract clinical text. In addition, semantic entanglement persists, where coarse-grained text embeddings blur the boundary between anatomical structures and imaging styles, thus weakening controllability during generation. To address this, we propose a Visually-Guided Text Disentanglement framework. We introduce a cross-modal latent alignment mechanism that leverages visual priors to explicitly disentangle unstructured text into independent semantic representations. Subsequently, a Hybrid Feature Fusion Module (HFFM) injects these features into a Diffusion Transformer (DiT) through separated channels, enabling fine-grained structural control. Experimental results in three datasets demonstrate that our method outperforms existing approaches in terms of generation quality and significantly improves performance on downstream classification tasks. The source code is available at https://github.com/hx111/VG-MedGen. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_10519 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement Huang, Xin Liang, Junjie Hou, Qingshan Cao, Peng Yang, Jinzhu Liu, Xiaoli Zaiane, Osmar R. Computer Vision and Pattern Recognition Medical image synthesis is crucial for alleviating data scarcity and privacy constraints. However, fine-tuning general text-to-image (T2I) models remains challenging, mainly due to the significant modality gap between complex visual details and abstract clinical text. In addition, semantic entanglement persists, where coarse-grained text embeddings blur the boundary between anatomical structures and imaging styles, thus weakening controllability during generation. To address this, we propose a Visually-Guided Text Disentanglement framework. We introduce a cross-modal latent alignment mechanism that leverages visual priors to explicitly disentangle unstructured text into independent semantic representations. Subsequently, a Hybrid Feature Fusion Module (HFFM) injects these features into a Diffusion Transformer (DiT) through separated channels, enabling fine-grained structural control. Experimental results in three datasets demonstrate that our method outperforms existing approaches in terms of generation quality and significantly improves performance on downstream classification tasks. The source code is available at https://github.com/hx111/VG-MedGen. |
| title | Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.10519 |