Safe Bimanual Teleoperation with Language-Guided Collision Avoidance
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913929775546368 |
|---|---|
| author | Totsila, Dionis Donoso, Clemente Hoffman, Enrico Mingo Mouret, Jean-Baptiste Ivaldi, Serena |
| author_facet | Totsila, Dionis Donoso, Clemente Hoffman, Enrico Mingo Mouret, Jean-Baptiste Ivaldi, Serena |
| contents | Teleoperating precise bimanual manipulations in cluttered environments is challenging for operators, who often struggle with limited spatial perception and difficulty estimating distances between target objects, the robot's body, obstacles, and the surrounding environment. To address these challenges, local robot perception and control should assist the operator during teleoperation. In this work, we introduce a safe teleoperation system that enhances operator control by preventing collisions in cluttered environments through the combination of immersive VR control and voice-activated collision avoidance. Using HTC Vive controllers, operators directly control a bimanual mobile manipulator, while spoken commands such as "avoid the yellow tool" trigger visual grounding and segmentation to build 3D obstacle meshes. These meshes are integrated into a whole-body controller to actively prevent collisions during teleoperation. Experiments in static, cluttered scenes demonstrate that our system significantly improves operational safety without compromising task efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04791 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Safe Bimanual Teleoperation with Language-Guided Collision Avoidance Totsila, Dionis Donoso, Clemente Hoffman, Enrico Mingo Mouret, Jean-Baptiste Ivaldi, Serena Robotics Teleoperating precise bimanual manipulations in cluttered environments is challenging for operators, who often struggle with limited spatial perception and difficulty estimating distances between target objects, the robot's body, obstacles, and the surrounding environment. To address these challenges, local robot perception and control should assist the operator during teleoperation. In this work, we introduce a safe teleoperation system that enhances operator control by preventing collisions in cluttered environments through the combination of immersive VR control and voice-activated collision avoidance. Using HTC Vive controllers, operators directly control a bimanual mobile manipulator, while spoken commands such as "avoid the yellow tool" trigger visual grounding and segmentation to build 3D obstacle meshes. These meshes are integrated into a whole-body controller to actively prevent collisions during teleoperation. Experiments in static, cluttered scenes demonstrate that our system significantly improves operational safety without compromising task efficiency. |
| title | Safe Bimanual Teleoperation with Language-Guided Collision Avoidance |
| topic | Robotics |
| url | https://arxiv.org/abs/2507.04791 |