InTraGen: Trajectory-controlled Video Generation for Object Interactions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918424062459904 |
|---|---|
| author | Liu, Zuhao Yanev, Aleksandar Mahmood, Ahmad Nikolov, Ivan Motamed, Saman Zheng, Wei-Shi Wang, Xi Sun, Lei Van Gool, Luc Paudel, Danda Pani |
| author_facet | Liu, Zuhao Yanev, Aleksandar Mahmood, Ahmad Nikolov, Ivan Motamed, Saman Zheng, Wei-Shi Wang, Xi Sun, Lei Van Gool, Luc Paudel, Danda Pani |
| contents | Advances in video generation have significantly improved the realism and quality of created scenes. This has fueled interest in developing intuitive tools that let users leverage video generation as world simulators. Text-to-video (T2V) generation is one such approach, enabling video creation from text descriptions only. Yet, due to the inherent ambiguity in texts and the limited temporal information offered by text prompts, researchers have explored additional control signals like trajectory-guided systems, for more accurate T2V generation. Nonetheless, methods to evaluate whether T2V models can generate realistic interactions between multiple objects are lacking. We introduce InTraGen, a pipeline for improved trajectory-based generation of object interaction scenarios. We propose 4 new datasets and a novel trajectory quality metric to evaluate the performance of the proposed InTraGen. To achieve object interaction, we introduce a multi-modal interaction encoding pipeline with an object ID injection mechanism that enriches object-environment interactions. Our results demonstrate improvements in both visual fidelity and quantitative performance. Code and datasets are available at https://github.com/insait-institute/InTraGen |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16804 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InTraGen: Trajectory-controlled Video Generation for Object Interactions Liu, Zuhao Yanev, Aleksandar Mahmood, Ahmad Nikolov, Ivan Motamed, Saman Zheng, Wei-Shi Wang, Xi Sun, Lei Van Gool, Luc Paudel, Danda Pani Computer Vision and Pattern Recognition Advances in video generation have significantly improved the realism and quality of created scenes. This has fueled interest in developing intuitive tools that let users leverage video generation as world simulators. Text-to-video (T2V) generation is one such approach, enabling video creation from text descriptions only. Yet, due to the inherent ambiguity in texts and the limited temporal information offered by text prompts, researchers have explored additional control signals like trajectory-guided systems, for more accurate T2V generation. Nonetheless, methods to evaluate whether T2V models can generate realistic interactions between multiple objects are lacking. We introduce InTraGen, a pipeline for improved trajectory-based generation of object interaction scenarios. We propose 4 new datasets and a novel trajectory quality metric to evaluate the performance of the proposed InTraGen. To achieve object interaction, we introduce a multi-modal interaction encoding pipeline with an object ID injection mechanism that enriches object-environment interactions. Our results demonstrate improvements in both visual fidelity and quantitative performance. Code and datasets are available at https://github.com/insait-institute/InTraGen |
| title | InTraGen: Trajectory-controlled Video Generation for Object Interactions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.16804 |