Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.17147 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911604568752128 |
|---|---|
| author | Gao, Lili Xu, Yanbo Koch, William Ruffino, Samuele Rowe, Luke Chalaki, Behdad Rivkin, Dmitriy Ost, Julian Girgis, Roger Bijelic, Mario Heide, Felix |
| author_facet | Gao, Lili Xu, Yanbo Koch, William Ruffino, Samuele Rowe, Luke Chalaki, Behdad Rivkin, Dmitriy Ost, Julian Girgis, Roger Bijelic, Mario Heide, Felix |
| contents | We introduce ScenarioControl, the first vision-language control mechanism for learned driving scenario generation. Given a text prompt or an input image, Scenario-Control synthesizes diverse, realistic 3D scenario rollouts - including map, 3D boxes of reactive actors over time, pedestrians, driving infrastructure, and ego camera observations. The method generates scenes in a vectorized latent space that represents road structure and dynamic agents jointly. To connect multimodal control with sparse vectorized scene elements, we propose a cross-global control mechanism that integrates crossattention with a lightweight global-context branch, enabling fine-grained control over road layout and traffic conditions while preserving realism. The method produces temporally consistent scenario rollouts from the perspectives different actors in the scene, supporting long-horizon continuation of driving scenarios. To facilitate training and evaluation, we release a dataset with text annotations aligned to vectorized map structures. Extensive experiments validate that the control adherence and fidelity of ScenarioControl compare favorable to all tested methods across all experiments. Project webpage: https://light.princeton.edu/ScenarioControl |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_17147 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation Gao, Lili Xu, Yanbo Koch, William Ruffino, Samuele Rowe, Luke Chalaki, Behdad Rivkin, Dmitriy Ost, Julian Girgis, Roger Bijelic, Mario Heide, Felix Computer Vision and Pattern Recognition Robotics We introduce ScenarioControl, the first vision-language control mechanism for learned driving scenario generation. Given a text prompt or an input image, Scenario-Control synthesizes diverse, realistic 3D scenario rollouts - including map, 3D boxes of reactive actors over time, pedestrians, driving infrastructure, and ego camera observations. The method generates scenes in a vectorized latent space that represents road structure and dynamic agents jointly. To connect multimodal control with sparse vectorized scene elements, we propose a cross-global control mechanism that integrates crossattention with a lightweight global-context branch, enabling fine-grained control over road layout and traffic conditions while preserving realism. The method produces temporally consistent scenario rollouts from the perspectives different actors in the scene, supporting long-horizon continuation of driving scenarios. To facilitate training and evaluation, we release a dataset with text annotations aligned to vectorized map structures. Extensive experiments validate that the control adherence and fidelity of ScenarioControl compare favorable to all tested methods across all experiments. Project webpage: https://light.princeton.edu/ScenarioControl |
| title | ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2604.17147 |