V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918115544137728 |
|---|---|
| author | Zhou, Zewei Xiang, Hao Zheng, Zhaoliang Zhao, Seth Z. Lei, Mingyue Zhang, Yun Cai, Tianhui Liu, Xinyi Liu, Johnson Bajji, Maheswari Xia, Xin Huang, Zhiyu Zhou, Bolei Ma, Jiaqi |
| author_facet | Zhou, Zewei Xiang, Hao Zheng, Zhaoliang Zhao, Seth Z. Lei, Mingyue Zhang, Yun Cai, Tianhui Liu, Xinyi Liu, Johnson Bajji, Maheswari Xia, Xin Huang, Zhiyu Zhou, Bolei Ma, Jiaqi |
| contents | Vehicle-to-everything (V2X) technologies offer a promising paradigm to mitigate the limitations of constrained observability in single-vehicle systems. Prior work primarily focuses on single-frame cooperative perception, which fuses agents' information across different spatial locations but ignores temporal cues and temporal tasks (e.g., temporal perception and prediction). In this paper, we focus on the spatio-temporal fusion in V2X scenarios and design one-step and multi-step communication strategies (when to transmit) as well as examine their integration with three fusion strategies - early, late, and intermediate (what to transmit), providing comprehensive benchmarks with 11 fusion models (how to fuse). Furthermore, we propose V2XPnP, a novel intermediate fusion framework within one-step communication for end-to-end perception and prediction. Our framework employs a unified Transformer-based architecture to effectively model complex spatio-temporal relationships across multiple agents, frames, and high-definition maps. Moreover, we introduce the V2XPnP Sequential Dataset that supports all V2X collaboration modes and addresses the limitations of existing real-world datasets, which are restricted to single-frame or single-mode cooperation. Extensive experiments demonstrate that our framework outperforms state-of-the-art methods in both perception and prediction tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01812 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction Zhou, Zewei Xiang, Hao Zheng, Zhaoliang Zhao, Seth Z. Lei, Mingyue Zhang, Yun Cai, Tianhui Liu, Xinyi Liu, Johnson Bajji, Maheswari Xia, Xin Huang, Zhiyu Zhou, Bolei Ma, Jiaqi Computer Vision and Pattern Recognition Vehicle-to-everything (V2X) technologies offer a promising paradigm to mitigate the limitations of constrained observability in single-vehicle systems. Prior work primarily focuses on single-frame cooperative perception, which fuses agents' information across different spatial locations but ignores temporal cues and temporal tasks (e.g., temporal perception and prediction). In this paper, we focus on the spatio-temporal fusion in V2X scenarios and design one-step and multi-step communication strategies (when to transmit) as well as examine their integration with three fusion strategies - early, late, and intermediate (what to transmit), providing comprehensive benchmarks with 11 fusion models (how to fuse). Furthermore, we propose V2XPnP, a novel intermediate fusion framework within one-step communication for end-to-end perception and prediction. Our framework employs a unified Transformer-based architecture to effectively model complex spatio-temporal relationships across multiple agents, frames, and high-definition maps. Moreover, we introduce the V2XPnP Sequential Dataset that supports all V2X collaboration modes and addresses the limitations of existing real-world datasets, which are restricted to single-frame or single-mode cooperation. Extensive experiments demonstrate that our framework outperforms state-of-the-art methods in both perception and prediction tasks. |
| title | V2XPnP: Vehicle-to-Everything Spatio-Temporal Fusion for Multi-Agent Perception and Prediction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.01812 |