Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.19603 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917624858804224 |
|---|---|
| author | Li, Chengzu Zhang, Chao Teufel, Simone Doddipatla, Rama Sanand Stoyanchev, Svetlana |
| author_facet | Li, Chengzu Zhang, Chao Teufel, Simone Doddipatla, Rama Sanand Stoyanchev, Svetlana |
| contents | We are interested in the generation of navigation instructions, either in their own right or as training material for robotic navigation task. In this paper, we propose a new approach to navigation instruction generation by framing the problem as an image captioning task using semantic maps as visual input. Conventional approaches employ a sequence of panorama images to generate navigation instructions. Semantic maps abstract away from visual details and fuse the information in multiple panorama images into a single top-down representation, thereby reducing computational complexity to process the input. We present a benchmark dataset for instruction generation using semantic maps, propose an initial model and ask human subjects to manually assess the quality of generated instructions. Our initial investigations show promise in using semantic maps for instruction generation instead of a sequence of panorama images, but there is vast scope for improvement. We release the code for data preparation and model training at https://github.com/chengzu-li/VLGen. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_19603 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Semantic Map-based Generation of Navigation Instructions Li, Chengzu Zhang, Chao Teufel, Simone Doddipatla, Rama Sanand Stoyanchev, Svetlana Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition We are interested in the generation of navigation instructions, either in their own right or as training material for robotic navigation task. In this paper, we propose a new approach to navigation instruction generation by framing the problem as an image captioning task using semantic maps as visual input. Conventional approaches employ a sequence of panorama images to generate navigation instructions. Semantic maps abstract away from visual details and fuse the information in multiple panorama images into a single top-down representation, thereby reducing computational complexity to process the input. We present a benchmark dataset for instruction generation using semantic maps, propose an initial model and ask human subjects to manually assess the quality of generated instructions. Our initial investigations show promise in using semantic maps for instruction generation instead of a sequence of panorama images, but there is vast scope for improvement. We release the code for data preparation and model training at https://github.com/chengzu-li/VLGen. |
| title | Semantic Map-based Generation of Navigation Instructions |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.19603 |