Matrix-3D: Omnidirectional Explorable 3D World Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909732516659200 |
|---|---|
| author | Yang, Zhongqi Ge, Wenhang Li, Yuqi Chen, Jiaqi Li, Haoyuan An, Mengyin Kang, Fei Xue, Hua Xu, Baixin Yin, Yuyang Li, Eric Liu, Yang Wang, Yikai Guo, Hao-Xiang Zhou, Yahui |
| author_facet | Yang, Zhongqi Ge, Wenhang Li, Yuqi Chen, Jiaqi Li, Haoyuan An, Mengyin Kang, Fei Xue, Hua Xu, Baixin Yin, Yuyang Li, Eric Liu, Yang Wang, Yikai Guo, Hao-Xiang Zhou, Yahui |
| contents | Explorable 3D world generation from a single image or text prompt forms a cornerstone of spatial intelligence. Recent works utilize video model to achieve wide-scope and generalizable 3D world generation. However, existing approaches often suffer from a limited scope in the generated scenes. In this work, we propose Matrix-3D, a framework that utilize panoramic representation for wide-coverage omnidirectional explorable 3D world generation that combines conditional video generation and panoramic 3D reconstruction. We first train a trajectory-guided panoramic video diffusion model that employs scene mesh renders as condition, to enable high-quality and geometrically consistent scene video generation. To lift the panorama scene video to 3D world, we propose two separate methods: (1) a feed-forward large panorama reconstruction model for rapid 3D scene reconstruction and (2) an optimization-based pipeline for accurate and detailed 3D scene reconstruction. To facilitate effective training, we also introduce the Matrix-Pano dataset, the first large-scale synthetic collection comprising 116K high-quality static panoramic video sequences with depth and trajectory annotations. Extensive experiments demonstrate that our proposed framework achieves state-of-the-art performance in panoramic video generation and 3D world generation. See more in https://matrix-3d.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_08086 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Matrix-3D: Omnidirectional Explorable 3D World Generation Yang, Zhongqi Ge, Wenhang Li, Yuqi Chen, Jiaqi Li, Haoyuan An, Mengyin Kang, Fei Xue, Hua Xu, Baixin Yin, Yuyang Li, Eric Liu, Yang Wang, Yikai Guo, Hao-Xiang Zhou, Yahui Computer Vision and Pattern Recognition Graphics Explorable 3D world generation from a single image or text prompt forms a cornerstone of spatial intelligence. Recent works utilize video model to achieve wide-scope and generalizable 3D world generation. However, existing approaches often suffer from a limited scope in the generated scenes. In this work, we propose Matrix-3D, a framework that utilize panoramic representation for wide-coverage omnidirectional explorable 3D world generation that combines conditional video generation and panoramic 3D reconstruction. We first train a trajectory-guided panoramic video diffusion model that employs scene mesh renders as condition, to enable high-quality and geometrically consistent scene video generation. To lift the panorama scene video to 3D world, we propose two separate methods: (1) a feed-forward large panorama reconstruction model for rapid 3D scene reconstruction and (2) an optimization-based pipeline for accurate and detailed 3D scene reconstruction. To facilitate effective training, we also introduce the Matrix-Pano dataset, the first large-scale synthetic collection comprising 116K high-quality static panoramic video sequences with depth and trajectory annotations. Extensive experiments demonstrate that our proposed framework achieves state-of-the-art performance in panoramic video generation and 3D world generation. See more in https://matrix-3d.github.io. |
| title | Matrix-3D: Omnidirectional Explorable 3D World Generation |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2508.08086 |