Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908852275904512 |
|---|---|
| author | Hu, JiaKui Liu, Jialun Yang, Liying Zhang, Xinliang Li, Kaiwen Zeng, Shuang Li, Yuanwei Huang, Haibin Zhang, Chi Lu, Yanye |
| author_facet | Hu, JiaKui Liu, Jialun Yang, Liying Zhang, Xinliang Li, Kaiwen Zeng, Shuang Li, Yuanwei Huang, Haibin Zhang, Chi Lu, Yanye |
| contents | Scene-consistent video generation aims to create videos that explore 3D scenes based on a camera trajectory. Previous methods rely on video generation models with external memory for consistency, or iterative 3D reconstruction and inpainting, which accumulate errors during inference due to incorrect intermediary outputs, non-differentiable processes, and separate models. To overcome these limitations, we introduce ``geometry-as-context". It iteratively completes the following steps using an autoregressive camera-controlled video generation model: (1) estimates the geometry of the current view necessary for 3D reconstruction, and (2) simulates and restores novel view images rendered by the 3D scene. Under this multi-task framework, we develop the camera gated attention module to enhance the model's capability to effectively leverage camera poses. During the training phase, text contexts are utilized to ascertain whether geometric or RGB images should be generated. To ensure that the model can generate RGB-only outputs during inference, the geometry context is randomly dropped from the interleaved text-image-geometry training sequence. The method has been tested on scene video generation with one-direction and forth-and-back trajectories. The results show its superiority over previous approaches in maintaining scene consistency and camera control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_21929 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context Hu, JiaKui Liu, Jialun Yang, Liying Zhang, Xinliang Li, Kaiwen Zeng, Shuang Li, Yuanwei Huang, Haibin Zhang, Chi Lu, Yanye Computer Vision and Pattern Recognition Scene-consistent video generation aims to create videos that explore 3D scenes based on a camera trajectory. Previous methods rely on video generation models with external memory for consistency, or iterative 3D reconstruction and inpainting, which accumulate errors during inference due to incorrect intermediary outputs, non-differentiable processes, and separate models. To overcome these limitations, we introduce ``geometry-as-context". It iteratively completes the following steps using an autoregressive camera-controlled video generation model: (1) estimates the geometry of the current view necessary for 3D reconstruction, and (2) simulates and restores novel view images rendered by the 3D scene. Under this multi-task framework, we develop the camera gated attention module to enhance the model's capability to effectively leverage camera poses. During the training phase, text contexts are utilized to ascertain whether geometric or RGB images should be generated. To ensure that the model can generate RGB-only outputs during inference, the geometry context is randomly dropped from the interleaved text-image-geometry training sequence. The method has been tested on scene video generation with one-direction and forth-and-back trajectories. The results show its superiority over previous approaches in maintaining scene consistency and camera control. |
| title | Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.21929 |