Tackling Structural Hallucination in Image Translation with Local Diffusion
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910530023718912 |
|---|---|
| author | Kim, Seunghoi Jin, Chen Diethe, Tom Figini, Matteo Tregidgo, Henry F. J. Mullokandov, Asher Teare, Philip Alexander, Daniel C. |
| author_facet | Kim, Seunghoi Jin, Chen Diethe, Tom Figini, Matteo Tregidgo, Henry F. J. Mullokandov, Asher Teare, Philip Alexander, Daniel C. |
| contents | Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing "image hallucination" and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local Diffusion processes. Our approach involves OOD estimation followed by two modules: a "branching" module generates locally both within and outside OOD regions, and a "fusion" module integrates these predictions into one. Our evaluation shows our method mitigates hallucination over baseline models quantitatively and qualitatively, reducing misdiagnosis by 40% and 25% in the real-world medical and natural image datasets, respectively. It also demonstrates compatibility with various pre-trained diffusion models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_05980 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tackling Structural Hallucination in Image Translation with Local Diffusion Kim, Seunghoi Jin, Chen Diethe, Tom Figini, Matteo Tregidgo, Henry F. J. Mullokandov, Asher Teare, Philip Alexander, Daniel C. Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Recent developments in diffusion models have advanced conditioned image generation, yet they struggle with reconstructing out-of-distribution (OOD) images, such as unseen tumors in medical images, causing "image hallucination" and risking misdiagnosis. We hypothesize such hallucinations result from local OOD regions in the conditional images. We verify that partitioning the OOD region and conducting separate image generations alleviates hallucinations in several applications. From this, we propose a training-free diffusion framework that reduces hallucination with multiple Local Diffusion processes. Our approach involves OOD estimation followed by two modules: a "branching" module generates locally both within and outside OOD regions, and a "fusion" module integrates these predictions into one. Our evaluation shows our method mitigates hallucination over baseline models quantitatively and qualitatively, reducing misdiagnosis by 40% and 25% in the real-world medical and natural image datasets, respectively. It also demonstrates compatibility with various pre-trained diffusion models. |
| title | Tackling Structural Hallucination in Image Translation with Local Diffusion |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2404.05980 |