LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917423817424896 |
|---|---|
| author | Tang, Jiaqi Xia, Yu Wu, Yi-Feng Hu, Yuwei Chen, Yuhui Chen, Qing-Guo Xu, Xiaogang Wu, Xiangyu Lu, Hao Ma, Yanqing Lu, Shiyin Chen, Qifeng |
| author_facet | Tang, Jiaqi Xia, Yu Wu, Yi-Feng Hu, Yuwei Chen, Yuhui Chen, Qing-Guo Xu, Xiaogang Wu, Xiangyu Lu, Hao Ma, Yanqing Lu, Shiyin Chen, Qifeng |
| contents | The advent of autonomous agents is transforming interactions with Graphical User Interfaces (GUIs) by employing natural language as a powerful intermediary. Despite the predominance of Supervised Fine-Tuning (SFT) methods in current GUI agents for achieving spatial localization, these methods face substantial challenges due to their limited capacity to accurately perceive positional data. Existing strategies, such as reinforcement learning, often fail to assess positional accuracy effectively, thereby restricting their utility. In response, we introduce Location Preference Optimization (LPO), a novel approach that leverages locational data to optimize interaction preferences. LPO uses information entropy to predict interaction positions by focusing on zones rich in information. Besides, it further introduces a dynamic location reward function based on physical distance, reflecting the varying importance of interaction positions. Supported by Group Relative Preference Optimization (GRPO), LPO facilitates an extensive exploration of GUI environments and significantly enhances interaction precision. Comprehensive experiments demonstrate LPO's superior performance, achieving SOTA results across both offline benchmarks and real-world online evaluations. Our code will be made publicly available soon, at https://github.com/jqtangust/LPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09373 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization Tang, Jiaqi Xia, Yu Wu, Yi-Feng Hu, Yuwei Chen, Yuhui Chen, Qing-Guo Xu, Xiaogang Wu, Xiangyu Lu, Hao Ma, Yanqing Lu, Shiyin Chen, Qifeng Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition The advent of autonomous agents is transforming interactions with Graphical User Interfaces (GUIs) by employing natural language as a powerful intermediary. Despite the predominance of Supervised Fine-Tuning (SFT) methods in current GUI agents for achieving spatial localization, these methods face substantial challenges due to their limited capacity to accurately perceive positional data. Existing strategies, such as reinforcement learning, often fail to assess positional accuracy effectively, thereby restricting their utility. In response, we introduce Location Preference Optimization (LPO), a novel approach that leverages locational data to optimize interaction preferences. LPO uses information entropy to predict interaction positions by focusing on zones rich in information. Besides, it further introduces a dynamic location reward function based on physical distance, reflecting the varying importance of interaction positions. Supported by Group Relative Preference Optimization (GRPO), LPO facilitates an extensive exploration of GUI environments and significantly enhances interaction precision. Comprehensive experiments demonstrate LPO's superior performance, achieving SOTA results across both offline benchmarks and real-world online evaluations. Our code will be made publicly available soon, at https://github.com/jqtangust/LPO. |
| title | LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.09373 |