ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910895141027840 |
|---|---|
| author | Jin, Yiqiao Petrangeli, Stefano Shen, Yu Wu, Gang |
| author_facet | Jin, Yiqiao Petrangeli, Stefano Shen, Yu Wu, Gang |
| contents | Graphical User Interface (GUI) agents are autonomous systems that interpret and generate actions, enabling intelligent user assistance and automation. Effective training of these agent presents unique challenges, such as sparsity in supervision signals, scalability for large datasets, and the need for nuanced user understanding. We propose stateful screen schema, an efficient representation of GUI interactions that captures key user actions and intentions over time. Building on this foundation, we introduce ScreenLLM, a set of multimodal large language models (MLLMs) tailored for advanced UI understanding and action prediction. Extensive experiments on both open-source and proprietary models show that ScreenLLM accurately models user behavior and predicts actions. Our work lays the foundation for scalable, robust, and intelligent GUI agents that enhance user interaction in diverse software environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_20978 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction Jin, Yiqiao Petrangeli, Stefano Shen, Yu Wu, Gang Computation and Language Graphical User Interface (GUI) agents are autonomous systems that interpret and generate actions, enabling intelligent user assistance and automation. Effective training of these agent presents unique challenges, such as sparsity in supervision signals, scalability for large datasets, and the need for nuanced user understanding. We propose stateful screen schema, an efficient representation of GUI interactions that captures key user actions and intentions over time. Building on this foundation, we introduce ScreenLLM, a set of multimodal large language models (MLLMs) tailored for advanced UI understanding and action prediction. Extensive experiments on both open-source and proprietary models show that ScreenLLM accurately models user behavior and predicts actions. Our work lays the foundation for scalable, robust, and intelligent GUI agents that enhance user interaction in diverse software environments. |
| title | ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.20978 |