Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.06413 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908420548853760 |
|---|---|
| author | Zhong, Yu Zhang, Rui Zhang, Zihao Wang, Shuo Fang, Chuan Zhang, Xishan Guo, Jiaming Peng, Shaohui Huang, Di Yan, Yanyang Hu, Xing Guo, Qi |
| author_facet | Zhong, Yu Zhang, Rui Zhang, Zihao Wang, Shuo Fang, Chuan Zhang, Xishan Guo, Jiaming Peng, Shaohui Huang, Di Yan, Yanyang Hu, Xing Guo, Qi |
| contents | Vision-and-Language Navigation (VLN) is a challenging task that requires an agent to navigate through photorealistic environments following natural-language instructions. One main obstacle existing in VLN is data scarcity, leading to poor generalization performance over unseen environments. Though data argumentation is a promising way for scaling up the dataset, how to generate VLN data both diverse and world-consistent remains problematic. To cope with this issue, we propose the world-consistent data generation (WCGEN), an efficacious data-augmentation framework satisfying both diversity and world-consistency, aimed at enhancing the generalization of agents to novel environments. Roughly, our framework consists of two stages, the trajectory stage which leverages a point-cloud based technique to ensure spatial coherency among viewpoints, and the viewpoint stage which adopts a novel angle synthesis method to guarantee spatial and wraparound consistency within the entire observation. By accurately predicting viewpoint changes with 3D knowledge, our approach maintains the world-consistency during the generation procedure. Experiments on a wide range of datasets verify the effectiveness of our method, demonstrating that our data augmentation strategy enables agents to achieve new state-of-the-art results on all navigation tasks, and is capable of enhancing the VLN agents' generalization ability to unseen environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06413 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | World-Consistent Data Generation for Vision-and-Language Navigation Zhong, Yu Zhang, Rui Zhang, Zihao Wang, Shuo Fang, Chuan Zhang, Xishan Guo, Jiaming Peng, Shaohui Huang, Di Yan, Yanyang Hu, Xing Guo, Qi Computer Vision and Pattern Recognition Vision-and-Language Navigation (VLN) is a challenging task that requires an agent to navigate through photorealistic environments following natural-language instructions. One main obstacle existing in VLN is data scarcity, leading to poor generalization performance over unseen environments. Though data argumentation is a promising way for scaling up the dataset, how to generate VLN data both diverse and world-consistent remains problematic. To cope with this issue, we propose the world-consistent data generation (WCGEN), an efficacious data-augmentation framework satisfying both diversity and world-consistency, aimed at enhancing the generalization of agents to novel environments. Roughly, our framework consists of two stages, the trajectory stage which leverages a point-cloud based technique to ensure spatial coherency among viewpoints, and the viewpoint stage which adopts a novel angle synthesis method to guarantee spatial and wraparound consistency within the entire observation. By accurately predicting viewpoint changes with 3D knowledge, our approach maintains the world-consistency during the generation procedure. Experiments on a wide range of datasets verify the effectiveness of our method, demonstrating that our data augmentation strategy enables agents to achieve new state-of-the-art results on all navigation tasks, and is capable of enhancing the VLN agents' generalization ability to unseen environments. |
| title | World-Consistent Data Generation for Vision-and-Language Navigation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.06413 |