Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.05838 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913979844001792 |
|---|---|
| author | Farooq, Ahmad Iqbal, Kamran |
| author_facet | Farooq, Ahmad Iqbal, Kamran |
| contents | This paper presents a novel approach that integrates vision foundation models with reinforcement learning to enhance object interaction capabilities in simulated environments. By combining the Segment Anything Model (SAM) and YOLOv5 with a Proximal Policy Optimization (PPO) agent operating in the AI2-THOR simulation environment, we enable the agent to perceive and interact with objects more effectively. Our comprehensive experiments, conducted across four diverse indoor kitchen settings, demonstrate significant improvements in object interaction success rates and navigation efficiency compared to a baseline agent without advanced perception. The results show a 68% increase in average cumulative reward, a 52.5% improvement in object interaction success rate, and a 33% increase in navigation efficiency. These findings highlight the potential of integrating foundation models with reinforcement learning for complex robotic tasks, paving the way for more sophisticated and capable autonomous agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05838 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Integrating Vision Foundation Models with Reinforcement Learning for Enhanced Object Interaction Farooq, Ahmad Iqbal, Kamran Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Systems and Control 68T07, 68T40, 90C40, 93E35 I.2.6; I.2.9; I.2.10 This paper presents a novel approach that integrates vision foundation models with reinforcement learning to enhance object interaction capabilities in simulated environments. By combining the Segment Anything Model (SAM) and YOLOv5 with a Proximal Policy Optimization (PPO) agent operating in the AI2-THOR simulation environment, we enable the agent to perceive and interact with objects more effectively. Our comprehensive experiments, conducted across four diverse indoor kitchen settings, demonstrate significant improvements in object interaction success rates and navigation efficiency compared to a baseline agent without advanced perception. The results show a 68% increase in average cumulative reward, a 52.5% improvement in object interaction success rate, and a 33% increase in navigation efficiency. These findings highlight the potential of integrating foundation models with reinforcement learning for complex robotic tasks, paving the way for more sophisticated and capable autonomous agents. |
| title | Integrating Vision Foundation Models with Reinforcement Learning for Enhanced Object Interaction |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Systems and Control 68T07, 68T40, 90C40, 93E35 I.2.6; I.2.9; I.2.10 |
| url | https://arxiv.org/abs/2508.05838 |