Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.12761 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929687186374656 |
|---|---|
| author | Brödermann, Tim Bruggemann, David Sakaridis, Christos Ta, Kevin Liagouris, Odysseas Corkill, Jason Van Gool, Luc |
| author_facet | Brödermann, Tim Bruggemann, David Sakaridis, Christos Ta, Kevin Liagouris, Odysseas Corkill, Jason Van Gool, Luc |
| contents | Achieving level-5 driving automation in autonomous vehicles necessitates a robust semantic visual perception system capable of parsing data from different sensors across diverse conditions. However, existing semantic perception datasets often lack important non-camera modalities typically used in autonomous vehicles, or they do not exploit such modalities to aid and improve semantic annotations in challenging conditions. To address this, we introduce MUSES, the MUlti-SEnsor Semantic perception dataset for driving in adverse conditions under increased uncertainty. MUSES includes synchronized multimodal recordings with 2D panoptic annotations for 2500 images captured under diverse weather and illumination. The dataset integrates a frame camera, a lidar, a radar, an event camera, and an IMU/GNSS sensor. Our new two-stage panoptic annotation protocol captures both class-level and instance-level uncertainty in the ground truth and enables the novel task of uncertainty-aware panoptic segmentation we introduce, along with standard semantic and panoptic segmentation. MUSES proves both effective for training and challenging for evaluating models under diverse visual conditions, and it opens new avenues for research in multimodal and uncertainty-aware dense semantic perception. Our dataset and benchmark are publicly available at https://muses.vision.ee.ethz.ch. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12761 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty Brödermann, Tim Bruggemann, David Sakaridis, Christos Ta, Kevin Liagouris, Odysseas Corkill, Jason Van Gool, Luc Computer Vision and Pattern Recognition Achieving level-5 driving automation in autonomous vehicles necessitates a robust semantic visual perception system capable of parsing data from different sensors across diverse conditions. However, existing semantic perception datasets often lack important non-camera modalities typically used in autonomous vehicles, or they do not exploit such modalities to aid and improve semantic annotations in challenging conditions. To address this, we introduce MUSES, the MUlti-SEnsor Semantic perception dataset for driving in adverse conditions under increased uncertainty. MUSES includes synchronized multimodal recordings with 2D panoptic annotations for 2500 images captured under diverse weather and illumination. The dataset integrates a frame camera, a lidar, a radar, an event camera, and an IMU/GNSS sensor. Our new two-stage panoptic annotation protocol captures both class-level and instance-level uncertainty in the ground truth and enables the novel task of uncertainty-aware panoptic segmentation we introduce, along with standard semantic and panoptic segmentation. MUSES proves both effective for training and challenging for evaluating models under diverse visual conditions, and it opens new avenues for research in multimodal and uncertainty-aware dense semantic perception. Our dataset and benchmark are publicly available at https://muses.vision.ee.ethz.ch. |
| title | MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.12761 |