Cover Image

Dynamic Planning for LLM-based Graphical User Interface Automation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Shaoqing, Zhang, Zhuosheng, Chen, Kehai, Ma, Xinbei, Yang, Muyun, Zhao, Tiejun, Zhang, Min
Format:	Preprint
Published:	2024
Subjects:	Artificial Intelligence Human-Computer Interaction
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM
by: Zhang, Shaoqing, et al.
Published: (2024)

GUIDE: Graphical User Interface Data for Execution
by: Chawla, Rajat, et al.
Published: (2024)

TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments
by: Lu, Yuheng, et al.
Published: (2025)

Visual Grounding Methods for Efficient Interaction with Desktop Graphical User Interfaces
by: Ettifouri, El Hassane, et al.
Published: (2024)

You Only Look at Screens: Multimodal Chain-of-Action Agents
by: Zhang, Zhuosheng, et al.
Published: (2023)

Human-Centered LLM-Agent User Interface: A Position Paper
by: Chin, Daniel, et al.
Published: (2024)

OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agents
by: Cheng, Pengzhou, et al.
Published: (2025)

How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors
by: Yu, Kuai, et al.
Published: (2026)

Navigating User Experience of ChatGPT-based Conversational Recommender Systems: The Effects of Prompt Guidance and Recommendation Domain
by: Zhang, Yizhe, et al.
Published: (2024)

The Command Line GUIde: Graphical Interfaces from Man Pages via AI
by: Kasibatla, Saketh Ram, et al.
Published: (2025)

MapAgent: Trajectory-Constructed Memory-Augmented Planning for Mobile Task Automation
by: Kong, Yi, et al.
Published: (2025)

Comparing LLM-Based Conversational and Graphical Interfaces for Industrial Decision Tasks: An Exploratory Mixed-Methods Study
by: Figliè, Roberto, et al.
Published: (2026)

MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents
by: Wang, Luyuan, et al.
Published: (2024)

Generative Interfaces for Language Models
by: Chen, Jiaqi, et al.
Published: (2025)

RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systems
by: Chen, Luyu, et al.
Published: (2025)

AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools
by: Riche, Nathalie, et al.
Published: (2025)

StoryVerse: Towards Co-authoring Dynamic Plot with LLM-based Character Simulation via Narrative Planning
by: Wang, Yi, et al.
Published: (2024)

A Deep User Interface for Exploring LLaMa
by: Perumal, Divya, et al.
Published: (2025)

Modeling User Preferences via Brain-Computer Interfacing
by: Leiva, Luis A., et al.
Published: (2024)

CellAgent: An LLM-driven Multi-Agent Framework for Automated Single-cell Data Analysis
by: Xiao, Yihang, et al.
Published: (2024)

MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces
by: Luera, Reuben A., et al.
Published: (2025)

Heuristics for AI-driven Graphical Asset Generation Tools in Game Design and Development Pipelines: A User-Centred Approach
by: Fukaya, Kaisei, et al.
Published: (2025)

Learning to Plan with Personalized Preferences
by: Xu, Manjie, et al.
Published: (2025)

Large Language Models Meet User Interfaces: The Case of Provisioning Feedback
by: Pozdniakov, Stanislav, et al.
Published: (2024)

AgentBuilder: Exploring Scaffolds for Prototyping User Experiences of Interface Agents
by: Liang, Jenny T., et al.
Published: (2025)

The AI-DEC: A Card-based Design Method for User-centered AI Explanations
by: Lee, Christine P, et al.
Published: (2024)

Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight
by: Tang, Jingyu, et al.
Published: (2025)

Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability
by: Bieniek, J., et al.
Published: (2024)

On Automating Video Game Regression Testing by Planning and Learning
by: Balyo, Tomáš, et al.
Published: (2024)

Explainable AI for Automated User-specific Feedback in Surgical Skill Acquisition
by: Gomez, Catalina, et al.
Published: (2025)

LLM4DESIGN: An Automated Multi-Modal System for Architectural and Environmental Design
by: Chen, Ran, et al.
Published: (2024)

Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces
by: Jiang, Yue, et al.
Published: (2024)

A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems
by: Zhu, Lixi, et al.
Published: (2024)

Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
by: Westhäußer, Rebecca, et al.
Published: (2025)

User Misconceptions of LLM-Based Conversational Programming Assistants
by: O'Brien, Gabrielle, et al.
Published: (2025)

Brief2Design: A Multi-phased, Compositional Approach to Prompt-based Graphic Design
by: Kikuchi, Kotaro, et al.
Published: (2026)

Frontend Diffusion: Exploring Intent-Based User Interfaces through Abstract-to-Detailed Task Transitions
by: Zhang, Qinshi, et al.
Published: (2024)

Navigating the Unknown: A Chat-Based Collaborative Interface for Personalized Exploratory Tasks
by: Peng, Yingzhe, et al.
Published: (2024)

Insights into User Interface Innovations from a Design Thinking Workshop at deRSE25
by: Frank, Maximilian, et al.
Published: (2025)

Toward User Comprehension Supports for LLM Agent Skill Specifications
by: Wen, Zikai Alex
Published: (2026)