Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.06230 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910823465615360 |
|---|---|
| author | Yao, Liang Liu, Fan Xu, Shengxiang Zhang, Chuanyi Ma, Xing Jiang, Jianyu Wang, Zequan Di, Shimin Zhou, Jun |
| author_facet | Yao, Liang Liu, Fan Xu, Shengxiang Zhang, Chuanyi Ma, Xing Jiang, Jianyu Wang, Zequan Di, Shimin Zhou, Jun |
| contents | The development of multi-modal learning for Unmanned Aerial Vehicles (UAVs) typically relies on a large amount of pixel-aligned multi-modal image data. However, existing datasets face challenges such as limited modalities, high construction costs, and imprecise annotations. To this end, we propose a synthetic multi-modal UAV-based multi-task dataset, UEMM-Air. Specifically, we simulate various UAV flight scenarios and object types using the Unreal Engine (UE). Then we design the UAV's flight logic to automatically collect data from different scenarios, perspectives, and altitudes. Furthermore, we propose a novel heuristic automatic annotation algorithm to generate accurate object detection labels. Finally, we utilize labels to generate text descriptions of images to make our UEMM-Air support more cross-modality tasks. In total, our UEMM-Air consists of 120k pairs of images with 6 modalities and precise annotations. Moreover, we conduct numerous experiments and establish new benchmark results on our dataset. We also found that models pre-trained on UEMM-Air exhibit better performance on downstream tasks compared to other similar datasets. The dataset is publicly available (https://github.com/1e12Leon/UEMM-Air) to support the research of multi-modal tasks on UAVs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06230 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UEMM-Air: Make Unmanned Aerial Vehicles Perform More Multi-modal Tasks Yao, Liang Liu, Fan Xu, Shengxiang Zhang, Chuanyi Ma, Xing Jiang, Jianyu Wang, Zequan Di, Shimin Zhou, Jun Computer Vision and Pattern Recognition The development of multi-modal learning for Unmanned Aerial Vehicles (UAVs) typically relies on a large amount of pixel-aligned multi-modal image data. However, existing datasets face challenges such as limited modalities, high construction costs, and imprecise annotations. To this end, we propose a synthetic multi-modal UAV-based multi-task dataset, UEMM-Air. Specifically, we simulate various UAV flight scenarios and object types using the Unreal Engine (UE). Then we design the UAV's flight logic to automatically collect data from different scenarios, perspectives, and altitudes. Furthermore, we propose a novel heuristic automatic annotation algorithm to generate accurate object detection labels. Finally, we utilize labels to generate text descriptions of images to make our UEMM-Air support more cross-modality tasks. In total, our UEMM-Air consists of 120k pairs of images with 6 modalities and precise annotations. Moreover, we conduct numerous experiments and establish new benchmark results on our dataset. We also found that models pre-trained on UEMM-Air exhibit better performance on downstream tasks compared to other similar datasets. The dataset is publicly available (https://github.com/1e12Leon/UEMM-Air) to support the research of multi-modal tasks on UAVs. |
| title | UEMM-Air: Make Unmanned Aerial Vehicles Perform More Multi-modal Tasks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.06230 |