DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908691496697856 |
|---|---|
| author | Wen, Kairun Huang, Yuzhi Chen, Runyu Zheng, Hui Lin, Yunlong Pan, Panwang Li, Chenxin Cong, Wenyan Zhang, Jian Lu, Junbin Lin, Chenguo Wang, Dilin Yan, Zhicheng Xu, Hongyu Theiss, Justin Huang, Yue Ding, Xinghao Ranjan, Rakesh Fan, Zhiwen |
| author_facet | Wen, Kairun Huang, Yuzhi Chen, Runyu Zheng, Hui Lin, Yunlong Pan, Panwang Li, Chenxin Cong, Wenyan Zhang, Jian Lu, Junbin Lin, Chenguo Wang, Dilin Yan, Zhicheng Xu, Hongyu Theiss, Justin Huang, Yue Ding, Xinghao Ranjan, Rakesh Fan, Zhiwen |
| contents | Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structurefrom-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_03000 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling Wen, Kairun Huang, Yuzhi Chen, Runyu Zheng, Hui Lin, Yunlong Pan, Panwang Li, Chenxin Cong, Wenyan Zhang, Jian Lu, Junbin Lin, Chenguo Wang, Dilin Yan, Zhicheng Xu, Hongyu Theiss, Justin Huang, Yue Ding, Xinghao Ranjan, Rakesh Fan, Zhiwen Computer Vision and Pattern Recognition Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structurefrom-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce DynamicVerse, a physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consisting of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods. |
| title | DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.03000 |