S2R-ViT for Multi-Agent Cooperative Perception: Bridging the Gap from Simulation to Reality
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909113870450688 |
|---|---|
| author | Li, Jinlong Xu, Runsheng Liu, Xinyu Li, Baolu Zou, Qin Ma, Jiaqi Yu, Hongkai |
| author_facet | Li, Jinlong Xu, Runsheng Liu, Xinyu Li, Baolu Zou, Qin Ma, Jiaqi Yu, Hongkai |
| contents | Due to the lack of enough real multi-agent data and time-consuming of labeling, existing multi-agent cooperative perception algorithms usually select the simulated sensor data for training and validating. However, the perception performance is degraded when these simulation-trained models are deployed to the real world, due to the significant domain gap between the simulated and real data. In this paper, we propose the first Simulation-to-Reality transfer learning framework for multi-agent cooperative perception using a novel Vision Transformer, named as S2R-ViT, which considers both the Deployment Gap and Feature Gap between simulated and real data. We investigate the effects of these two types of domain gaps and propose a novel uncertainty-aware vision transformer to effectively relief the Deployment Gap and an agent-based feature adaptation module with inter-agent and ego-agent discriminators to reduce the Feature Gap. Our intensive experiments on the public multi-agent cooperative perception datasets OPV2V and V2V4Real demonstrate that the proposed S2R-ViT can effectively bridge the gap from simulation to reality and outperform other methods significantly for point cloud-based 3D object detection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_07935 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | S2R-ViT for Multi-Agent Cooperative Perception: Bridging the Gap from Simulation to Reality Li, Jinlong Xu, Runsheng Liu, Xinyu Li, Baolu Zou, Qin Ma, Jiaqi Yu, Hongkai Computer Vision and Pattern Recognition Due to the lack of enough real multi-agent data and time-consuming of labeling, existing multi-agent cooperative perception algorithms usually select the simulated sensor data for training and validating. However, the perception performance is degraded when these simulation-trained models are deployed to the real world, due to the significant domain gap between the simulated and real data. In this paper, we propose the first Simulation-to-Reality transfer learning framework for multi-agent cooperative perception using a novel Vision Transformer, named as S2R-ViT, which considers both the Deployment Gap and Feature Gap between simulated and real data. We investigate the effects of these two types of domain gaps and propose a novel uncertainty-aware vision transformer to effectively relief the Deployment Gap and an agent-based feature adaptation module with inter-agent and ego-agent discriminators to reduce the Feature Gap. Our intensive experiments on the public multi-agent cooperative perception datasets OPV2V and V2V4Real demonstrate that the proposed S2R-ViT can effectively bridge the gap from simulation to reality and outperform other methods significantly for point cloud-based 3D object detection. |
| title | S2R-ViT for Multi-Agent Cooperative Perception: Bridging the Gap from Simulation to Reality |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2307.07935 |