Towards Language-Driven Video Inpainting via Multimodal Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912052391444480 |
|---|---|
| author | Wu, Jianzong Li, Xiangtai Si, Chenyang Zhou, Shangchen Yang, Jingkang Zhang, Jiangning Li, Yining Chen, Kai Tong, Yunhai Liu, Ziwei Loy, Chen Change |
| author_facet | Wu, Jianzong Li, Xiangtai Si, Chenyang Zhou, Shangchen Yang, Jingkang Zhang, Jiangning Li, Yining Chen, Kai Tong, Yunhai Liu, Ziwei Loy, Chen Change |
| contents | We introduce a new task -- language-driven video inpainting, which uses natural language instructions to guide the inpainting process. This approach overcomes the limitations of traditional video inpainting methods that depend on manually labeled binary masks, a process often tedious and labor-intensive. We present the Remove Objects from Videos by Instructions (ROVI) dataset, containing 5,650 videos and 9,091 inpainting results, to support training and evaluation for this task. We also propose a novel diffusion-based language-driven video inpainting framework, the first end-to-end baseline for this task, integrating Multimodal Large Language Models to understand and execute complex language-based inpainting requests effectively. Our comprehensive results showcase the dataset's versatility and the model's effectiveness in various language-instructed inpainting scenarios. We will make datasets, code, and models publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10226 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Language-Driven Video Inpainting via Multimodal Large Language Models Wu, Jianzong Li, Xiangtai Si, Chenyang Zhou, Shangchen Yang, Jingkang Zhang, Jiangning Li, Yining Chen, Kai Tong, Yunhai Liu, Ziwei Loy, Chen Change Computer Vision and Pattern Recognition We introduce a new task -- language-driven video inpainting, which uses natural language instructions to guide the inpainting process. This approach overcomes the limitations of traditional video inpainting methods that depend on manually labeled binary masks, a process often tedious and labor-intensive. We present the Remove Objects from Videos by Instructions (ROVI) dataset, containing 5,650 videos and 9,091 inpainting results, to support training and evaluation for this task. We also propose a novel diffusion-based language-driven video inpainting framework, the first end-to-end baseline for this task, integrating Multimodal Large Language Models to understand and execute complex language-based inpainting requests effectively. Our comprehensive results showcase the dataset's versatility and the model's effectiveness in various language-instructed inpainting scenarios. We will make datasets, code, and models publicly available. |
| title | Towards Language-Driven Video Inpainting via Multimodal Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.10226 |