Learning to See and Act: Task-Aware Virtual View Exploration for Robotic Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918394546094080 |
|---|---|
| author | Bai, Yongjie Wang, Zhouxia Liu, Yang Luo, Kaijun Wen, Yifan Dai, Mingtong Chen, Weixing Chen, Ziliang Liu, Lingbo Li, Guanbin Lin, Liang |
| author_facet | Bai, Yongjie Wang, Zhouxia Liu, Yang Luo, Kaijun Wen, Yifan Dai, Mingtong Chen, Weixing Chen, Ziliang Liu, Lingbo Li, Guanbin Lin, Liang |
| contents | Recent vision-language-action (VLA) models for multi-task robot manipulation often rely on fixed camera setups and shared visual encoders, which limit their performance under occlusions and during cross-task transfer. To address these challenges, we propose Task-aware Virtual View Exploration (TVVE), a framework that learns to select task-relevant virtual camera viewpoints and dynamically re-render observations from a reconstructed scene representation using the selected viewpoints. To enable efficient view selection, we train an exploration policy in a pseudo-environment. In addition, we introduce a Task-aware Mixture-of-Experts (TaskMoE) visual encoder that routes visual features to task-specialized experts, mitigating interference in multi-task learning. To evaluate robustness under distribution shifts, we construct RLBench-OG, an out-of-distribution benchmark with visual perturbations and camera pose variations. Experiments on RLBench and RLBench-OG demonstrate that TVVE achieves higher success rates than strong baselines, while real-robot experiments further confirm its robustness to visual disturbances and unseen instructions. Code and visualizations are available at: https://hcplab-sysu.github.io/TAVP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05186 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning to See and Act: Task-Aware Virtual View Exploration for Robotic Manipulation Bai, Yongjie Wang, Zhouxia Liu, Yang Luo, Kaijun Wen, Yifan Dai, Mingtong Chen, Weixing Chen, Ziliang Liu, Lingbo Li, Guanbin Lin, Liang Robotics Computer Vision and Pattern Recognition Recent vision-language-action (VLA) models for multi-task robot manipulation often rely on fixed camera setups and shared visual encoders, which limit their performance under occlusions and during cross-task transfer. To address these challenges, we propose Task-aware Virtual View Exploration (TVVE), a framework that learns to select task-relevant virtual camera viewpoints and dynamically re-render observations from a reconstructed scene representation using the selected viewpoints. To enable efficient view selection, we train an exploration policy in a pseudo-environment. In addition, we introduce a Task-aware Mixture-of-Experts (TaskMoE) visual encoder that routes visual features to task-specialized experts, mitigating interference in multi-task learning. To evaluate robustness under distribution shifts, we construct RLBench-OG, an out-of-distribution benchmark with visual perturbations and camera pose variations. Experiments on RLBench and RLBench-OG demonstrate that TVVE achieves higher success rates than strong baselines, while real-robot experiments further confirm its robustness to visual disturbances and unseen instructions. Code and visualizations are available at: https://hcplab-sysu.github.io/TAVP. |
| title | Learning to See and Act: Task-Aware Virtual View Exploration for Robotic Manipulation |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.05186 |