Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916798689968128 |
|---|---|
| author | Ren, Xuanchi Lu, Yifan Cao, Tianshi Gao, Ruiyuan Huang, Shengyu Sabour, Amirmojtaba Shen, Tianchang Pfaff, Tobias Wu, Jay Zhangjie Chen, Runjian Kim, Seung Wook Gao, Jun Leal-Taixe, Laura Chen, Mike Fidler, Sanja Ling, Huan |
| author_facet | Ren, Xuanchi Lu, Yifan Cao, Tianshi Gao, Ruiyuan Huang, Shengyu Sabour, Amirmojtaba Shen, Tianchang Pfaff, Tobias Wu, Jay Zhangjie Chen, Runjian Kim, Seung Wook Gao, Jun Leal-Taixe, Laura Chen, Mike Fidler, Sanja Ling, Huan |
| contents | Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform.
Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09042 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models Ren, Xuanchi Lu, Yifan Cao, Tianshi Gao, Ruiyuan Huang, Shengyu Sabour, Amirmojtaba Shen, Tianchang Pfaff, Tobias Wu, Jay Zhangjie Chen, Runjian Kim, Seung Wook Gao, Jun Leal-Taixe, Laura Chen, Mike Fidler, Sanja Ling, Huan Computer Vision and Pattern Recognition Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform. Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams |
| title | Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.09042 |