Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912810700636160 |
|---|---|
| author | Tay, Tracey Yee Hsin Yan, Xu Ouyang, Jonathan Wu, Daniel Jiang, William Kao, Jonathan Cui, Yuchen |
| author_facet | Tay, Tracey Yee Hsin Yan, Xu Ouyang, Jonathan Wu, Daniel Jiang, William Kao, Jonathan Cui, Yuchen |
| contents | Designing intuitive interfaces for robotic control remains a central challenge in enabling effective human-robot interaction, particularly in assistive care settings. Eye gaze offers a fast, non-intrusive, and intent-rich input modality, making it an attractive channel for conveying user goals. In this work, we present GAMMA (Gaze Assisted Manipulation for Modular Autonomy), a system that leverages ego-centric gaze tracking and a vision-language model to infer user intent and autonomously execute robotic manipulation tasks. By contextualizing gaze fixations within the scene, the system maps visual attention to high-level semantic understanding, enabling skill selection and parameterization without task-specific training. We evaluate GAMMA on a range of table-top manipulation tasks and compare it against baseline gaze-based control without reasoning. Results demonstrate that GAMMA provides robust, intuitive, and generalizable control, highlighting the potential of combining foundation models and gaze for natural and scalable robot autonomy. Project website: https://gamma0.vercel.app/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_05336 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models Tay, Tracey Yee Hsin Yan, Xu Ouyang, Jonathan Wu, Daniel Jiang, William Kao, Jonathan Cui, Yuchen Robotics Designing intuitive interfaces for robotic control remains a central challenge in enabling effective human-robot interaction, particularly in assistive care settings. Eye gaze offers a fast, non-intrusive, and intent-rich input modality, making it an attractive channel for conveying user goals. In this work, we present GAMMA (Gaze Assisted Manipulation for Modular Autonomy), a system that leverages ego-centric gaze tracking and a vision-language model to infer user intent and autonomously execute robotic manipulation tasks. By contextualizing gaze fixations within the scene, the system maps visual attention to high-level semantic understanding, enabling skill selection and parameterization without task-specific training. We evaluate GAMMA on a range of table-top manipulation tasks and compare it against baseline gaze-based control without reasoning. Results demonstrate that GAMMA provides robust, intuitive, and generalizable control, highlighting the potential of combining foundation models and gaze for natural and scalable robot autonomy. Project website: https://gamma0.vercel.app/ |
| title | Intent at a Glance: Gaze-Guided Robotic Manipulation via Foundation Models |
| topic | Robotics |
| url | https://arxiv.org/abs/2601.05336 |