Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908859144077312 |
|---|---|
| author | Wahl, Ari Gawlinski, Dorian Przewozny, David Chojecki, Paul Bießmann, Felix Bosse, Sebastian |
| author_facet | Wahl, Ari Gawlinski, Dorian Przewozny, David Chojecki, Paul Bießmann, Felix Bosse, Sebastian |
| contents | Pre-trained general-purpose Vision-Language Models (VLM) hold the potential to enhance intuitive human-machine interactions due to their rich world knowledge and 2D object detection capabilities. However, VLMs for 3D coordinates detection tasks are rare. In this work, we investigate interactive abilities of VLMs by returning 3D object positions given a monocular RGB image from a wrist-mounted camera, natural language input, and robot states. We collected and curated a heterogeneous dataset of more than 100,000 images and finetuned a VLM using QLoRA with a custom regression head. By implementing conditional routing, our model maintains its ability to process general visual queries while adding specialized 3D position estimation capabilities. Our results demonstrate robust predictive performance with a median MAE of 13 mm on the test set and a five-fold improvement over a simpler baseline without finetuning. In about 25% of the cases, predictions are within a range considered acceptable for the robot to interact with objects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01224 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction Wahl, Ari Gawlinski, Dorian Przewozny, David Chojecki, Paul Bießmann, Felix Bosse, Sebastian Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction Machine Learning Robotics Pre-trained general-purpose Vision-Language Models (VLM) hold the potential to enhance intuitive human-machine interactions due to their rich world knowledge and 2D object detection capabilities. However, VLMs for 3D coordinates detection tasks are rare. In this work, we investigate interactive abilities of VLMs by returning 3D object positions given a monocular RGB image from a wrist-mounted camera, natural language input, and robot states. We collected and curated a heterogeneous dataset of more than 100,000 images and finetuned a VLM using QLoRA with a custom regression head. By implementing conditional routing, our model maintains its ability to process general visual queries while adding specialized 3D position estimation capabilities. Our results demonstrate robust predictive performance with a median MAE of 13 mm on the test set and a five-fold improvement over a simpler baseline without finetuning. In about 25% of the cases, predictions are within a range considered acceptable for the robot to interact with objects. |
| title | Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction Machine Learning Robotics |
| url | https://arxiv.org/abs/2603.01224 |