Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2501.09499 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916568469864448 |
|---|---|
| author | Fang, Zixun Liu, Zhiheng Zhu, Kai Liu, Yu Cheng, Ka Leong Zhai, Wei Cao, Yang Zha, Zheng-Jun |
| author_facet | Fang, Zixun Liu, Zhiheng Zhu, Kai Liu, Yu Cheng, Ka Leong Zhai, Wei Cao, Yang Zha, Zheng-Jun |
| contents | Video colorization aims to transform grayscale videos into vivid color representations while maintaining temporal consistency and structural integrity. Existing video colorization methods often suffer from color bleeding and lack comprehensive control, particularly under complex motion or diverse semantic cues. To this end, we introduce VanGogh, a unified multimodal diffusion-based framework for video colorization. VanGogh tackles these challenges using a Dual Qformer to align and fuse features from multiple modalities, complemented by a depth-guided generation process and an optical flow loss, which help reduce color overflow. Additionally, a color injection strategy and luma channel replacement are implemented to improve generalization and mitigate flickering artifacts. Thanks to this design, users can exercise both global and local control over the generation process, resulting in higher-quality colorized videos. Extensive qualitative and quantitative evaluations, and user studies, demonstrate that VanGogh achieves superior temporal consistency and color fidelity.Project page: https://becauseimbatman0.github.io/VanGogh. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09499 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization Fang, Zixun Liu, Zhiheng Zhu, Kai Liu, Yu Cheng, Ka Leong Zhai, Wei Cao, Yang Zha, Zheng-Jun Computer Vision and Pattern Recognition Video colorization aims to transform grayscale videos into vivid color representations while maintaining temporal consistency and structural integrity. Existing video colorization methods often suffer from color bleeding and lack comprehensive control, particularly under complex motion or diverse semantic cues. To this end, we introduce VanGogh, a unified multimodal diffusion-based framework for video colorization. VanGogh tackles these challenges using a Dual Qformer to align and fuse features from multiple modalities, complemented by a depth-guided generation process and an optical flow loss, which help reduce color overflow. Additionally, a color injection strategy and luma channel replacement are implemented to improve generalization and mitigate flickering artifacts. Thanks to this design, users can exercise both global and local control over the generation process, resulting in higher-quality colorized videos. Extensive qualitative and quantitative evaluations, and user studies, demonstrate that VanGogh achieves superior temporal consistency and color fidelity.Project page: https://becauseimbatman0.github.io/VanGogh. |
| title | VanGogh: A Unified Multimodal Diffusion-based Framework for Video Colorization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.09499 |