Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.14227 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917759052414976 |
|---|---|
| author | Doan, Anh-Dzung Phan, Vu Minh Hieu Gupta, Surabhi Wagner, Markus Chin, Tat-Jun Reid, Ian |
| author_facet | Doan, Anh-Dzung Phan, Vu Minh Hieu Gupta, Surabhi Wagner, Markus Chin, Tat-Jun Reid, Ian |
| contents | Infrared imaging offers resilience against changing lighting conditions by capturing object temperatures. Yet, in few scenarios, its lack of visual details compared to daytime visible images, poses a significant challenge for human and machine interpretation. This paper proposes a novel diffusion method, dubbed Temporally Consistent Patch Diffusion Models (TC-DPM), for infrared-to-visible video translation. Our method, extending the Patch Diffusion Model, consists of two key components. Firstly, we propose a semantic-guided denoising, leveraging the strong representations of foundational models. As such, our method faithfully preserves the semantic structure of generated visible images. Secondly, we propose a novel temporal blending module to guide the denoising trajectory, ensuring the temporal consistency between consecutive frames. Experiment shows that TC-PDM outperforms state-of-the-art methods by 35.3% in FVD for infrared-to-visible video translation and by 6.1% in AP50 for day-to-night object detection. Our code is publicly available at https://github.com/dzungdoan6/tc-pdm |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_14227 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TC-PDM: Temporally Consistent Patch Diffusion Models for Infrared-to-Visible Video Translation Doan, Anh-Dzung Phan, Vu Minh Hieu Gupta, Surabhi Wagner, Markus Chin, Tat-Jun Reid, Ian Computer Vision and Pattern Recognition Infrared imaging offers resilience against changing lighting conditions by capturing object temperatures. Yet, in few scenarios, its lack of visual details compared to daytime visible images, poses a significant challenge for human and machine interpretation. This paper proposes a novel diffusion method, dubbed Temporally Consistent Patch Diffusion Models (TC-DPM), for infrared-to-visible video translation. Our method, extending the Patch Diffusion Model, consists of two key components. Firstly, we propose a semantic-guided denoising, leveraging the strong representations of foundational models. As such, our method faithfully preserves the semantic structure of generated visible images. Secondly, we propose a novel temporal blending module to guide the denoising trajectory, ensuring the temporal consistency between consecutive frames. Experiment shows that TC-PDM outperforms state-of-the-art methods by 35.3% in FVD for infrared-to-visible video translation and by 6.1% in AP50 for day-to-night object detection. Our code is publicly available at https://github.com/dzungdoan6/tc-pdm |
| title | TC-PDM: Temporally Consistent Patch Diffusion Models for Infrared-to-Visible Video Translation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.14227 |