Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908360795750400 |
|---|---|
| author | Chaichuk, Mikhail Gautam, Sushant Hicks, Steven Tutubalina, Elena |
| author_facet | Chaichuk, Mikhail Gautam, Sushant Hicks, Steven Tutubalina, Elena |
| contents | The generation of realistic medical images from text descriptions has significant potential to address data scarcity challenges in healthcare AI while preserving patient privacy. This paper presents a comprehensive study of text-to-image synthesis in the medical domain, comparing two distinct approaches: (1) fine-tuning large pre-trained latent diffusion models and (2) training small, domain-specific models. We introduce a novel model named MSDM, an optimized architecture based on Stable Diffusion that integrates a clinical text encoder, variational autoencoder, and cross-attention mechanisms to better align medical text prompts with generated images. Our study compares two approaches: fine-tuning large pre-trained models (FLUX, Kandinsky) versus training compact domain-specific models (MSDM). Evaluation across colonoscopy (MedVQA-GI) and radiology (ROCOv2) datasets reveals that while large models achieve higher fidelity, our optimized MSDM delivers comparable quality with lower computational costs. Quantitative metrics and qualitative evaluations by medical experts reveal strengths and limitations of each approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05573 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models Chaichuk, Mikhail Gautam, Sushant Hicks, Steven Tutubalina, Elena Computer Vision and Pattern Recognition Artificial Intelligence 68T07, 68U10, 92C55 I.2.10; I.4.8; J.3 The generation of realistic medical images from text descriptions has significant potential to address data scarcity challenges in healthcare AI while preserving patient privacy. This paper presents a comprehensive study of text-to-image synthesis in the medical domain, comparing two distinct approaches: (1) fine-tuning large pre-trained latent diffusion models and (2) training small, domain-specific models. We introduce a novel model named MSDM, an optimized architecture based on Stable Diffusion that integrates a clinical text encoder, variational autoencoder, and cross-attention mechanisms to better align medical text prompts with generated images. Our study compares two approaches: fine-tuning large pre-trained models (FLUX, Kandinsky) versus training compact domain-specific models (MSDM). Evaluation across colonoscopy (MedVQA-GI) and radiology (ROCOv2) datasets reveals that while large models achieve higher fidelity, our optimized MSDM delivers comparable quality with lower computational costs. Quantitative metrics and qualitative evaluations by medical experts reveal strengths and limitations of each approach. |
| title | Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence 68T07, 68U10, 92C55 I.2.10; I.4.8; J.3 |
| url | https://arxiv.org/abs/2505.05573 |