InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910743062904832 |
|---|---|
| author | Fan, Tiehan Nan, Kepan Xie, Rui Zhou, Penghao Yang, Zhenheng Fu, Chaoyou Li, Xiang Yang, Jian Tai, Ying |
| author_facet | Fan, Tiehan Nan, Kepan Xie, Rui Zhou, Penghao Yang, Zhenheng Fu, Chaoyou Li, Xiang Yang, Jian Tai, Ying |
| contents | Text-to-video generation has evolved rapidly in recent years, delivering remarkable results. Training typically relies on video-caption paired data, which plays a crucial role in enhancing generation performance. However, current video captions often suffer from insufficient details, hallucinations and imprecise motion depiction, affecting the fidelity and consistency of generated videos. In this work, we propose a novel instance-aware structured caption framework, termed InstanceCap, to achieve instance-level and fine-grained video caption for the first time. Based on this scheme, we design an auxiliary models cluster to convert original video into instances to enhance instance fidelity. Video instances are further used to refine dense prompts into structured phrases, achieving concise yet precise descriptions. Furthermore, a 22K InstanceVid dataset is curated for training, and an enhancement pipeline that tailored to InstanceCap structure is proposed for inference. Experimental results demonstrate that our proposed InstanceCap significantly outperform previous models, ensuring high fidelity between captions and videos while reducing hallucinations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09283 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption Fan, Tiehan Nan, Kepan Xie, Rui Zhou, Penghao Yang, Zhenheng Fu, Chaoyou Li, Xiang Yang, Jian Tai, Ying Computer Vision and Pattern Recognition Artificial Intelligence Text-to-video generation has evolved rapidly in recent years, delivering remarkable results. Training typically relies on video-caption paired data, which plays a crucial role in enhancing generation performance. However, current video captions often suffer from insufficient details, hallucinations and imprecise motion depiction, affecting the fidelity and consistency of generated videos. In this work, we propose a novel instance-aware structured caption framework, termed InstanceCap, to achieve instance-level and fine-grained video caption for the first time. Based on this scheme, we design an auxiliary models cluster to convert original video into instances to enhance instance fidelity. Video instances are further used to refine dense prompts into structured phrases, achieving concise yet precise descriptions. Furthermore, a 22K InstanceVid dataset is curated for training, and an enhancement pipeline that tailored to InstanceCap structure is proposed for inference. Experimental results demonstrate that our proposed InstanceCap significantly outperform previous models, ensuring high fidelity between captions and videos while reducing hallucinations. |
| title | InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2412.09283 |