Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.15425 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911011977560064 |
|---|---|
| author | Tao, Xingjian Wang, Yiwei Cai, Yujun Yang, Zhicheng Tang, Jing |
| author_facet | Tao, Xingjian Wang, Yiwei Cai, Yujun Yang, Zhicheng Tang, Jing |
| contents | Multimodal large language models (MLLMs) have enabled GUI agents to interact with operating systems by grounding language into spatial actions. Despite their promising performance, these models frequently exhibit hallucinations-systematic localization errors that compromise reliability. We propose a fine-grained evaluation framework that categorizes model predictions into four distinct types, revealing nuanced failure modes beyond traditional accuracy metrics. To better quantify model uncertainty, we introduce the Peak Sharpness Score (PSS), a metric that evaluates the alignment between semantic continuity and logits distribution in coordinate prediction. Building on this insight, we further propose Context-Aware Cropping, a training-free technique that improves model performance by adaptively refining input context. Extensive experiments demonstrate that our framework and methods provide actionable insights and enhance the interpretability and robustness of GUI agent behavior. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_15425 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understanding GUI Agent Localization Biases through Logit Sharpness Tao, Xingjian Wang, Yiwei Cai, Yujun Yang, Zhicheng Tang, Jing Computation and Language Multimodal large language models (MLLMs) have enabled GUI agents to interact with operating systems by grounding language into spatial actions. Despite their promising performance, these models frequently exhibit hallucinations-systematic localization errors that compromise reliability. We propose a fine-grained evaluation framework that categorizes model predictions into four distinct types, revealing nuanced failure modes beyond traditional accuracy metrics. To better quantify model uncertainty, we introduce the Peak Sharpness Score (PSS), a metric that evaluates the alignment between semantic continuity and logits distribution in coordinate prediction. Building on this insight, we further propose Context-Aware Cropping, a training-free technique that improves model performance by adaptively refining input context. Extensive experiments demonstrate that our framework and methods provide actionable insights and enhance the interpretability and robustness of GUI agent behavior. |
| title | Understanding GUI Agent Localization Biases through Logit Sharpness |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.15425 |