Cover Image

Agent-Initiated Interaction in Phone UI Automation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kahlon, Noam, Rom, Guy, Efros, Anatoly, Galgani, Filippo, Berkovitch, Omri, Caduri, Sapir, Bishop, William E., Riva, Oriana, Dagan, Ido
Format:	Preprint
Published:	2025
Subjects:	Human-Computer Interaction
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Identifying User Goals from UI Trajectories
by: Berkovitch, Omri, et al.
Published: (2024)

Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories
by: Caduri, Sapir, et al.
Published: (2025)

Small Models, Big Results: Achieving Superior Intent Extraction through Decomposition
by: Cohen, Danielle, et al.
Published: (2025)

UINav: A Practical Approach to Train On-Device Automation Agents
by: Li, Wei, et al.
Published: (2023)

Latent State Estimation Helps UI Agents to Reason
by: Bishop, William E, et al.
Published: (2024)

UI-TARS: Pioneering Automated GUI Interaction with Native Agents
by: Qin, Yujia, et al.
Published: (2025)

LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects
by: Liu, Guangyi, et al.
Published: (2025)

On the Effects of Data Scale on UI Control Agents
by: Li, Wei, et al.
Published: (2024)

GhostUI: Unveiling Hidden Interactions in Mobile UI
by: Kweon, Minkyu, et al.
Published: (2026)

UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing
by: Wang, Junling, et al.
Published: (2026)

Macaron-A2UI: A Model for Generative UI in Personal Agents
by: Kong, Fancy, et al.
Published: (2026)

MLLM-Based UI2Code Automation Guided by UI Layout Information
by: Wu, Fan, et al.
Published: (2025)

MAIC-UI: Making Interactive Courseware with Generative UI
by: Tu, Shangqing, et al.
Published: (2026)

Automating UI Optimization through Multi-Agentic Reasoning
by: Li, Zhipeng, et al.
Published: (2026)

UFO: A UI-Focused Agent for Windows OS Interaction
by: Zhang, Chaoyun, et al.
Published: (2024)

Misty: UI Prototyping Through Interactive Conceptual Blending
by: Lu, Yuwen, et al.
Published: (2024)

From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts
by: Zhang, Zhuohao Jerry, et al.
Published: (2024)

Automated UI Interface Generation via Diffusion Models: Enhancing Personalization and Efficiency
by: Duan, Yifei, et al.
Published: (2025)

Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts
by: Huang, Tian, et al.
Published: (2024)

Open WebUI: An Open, Extensible, and Usable Interface for AI Interaction
by: Baek, Jaeryang, et al.
Published: (2025)

ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent
by: Zhao, Yuheng, et al.
Published: (2025)

Optimizing Gesture Recognition for Seamless UI Interaction Using Convolutional Neural Networks
by: Sun, Qi, et al.
Published: (2024)

Privacy Starts with UI: Privacy Patterns and Designer Perspectives in UI/UX Practice
by: Maloku, Anxhela, et al.
Published: (2026)

Moving Phones, Active Peers: Exploring the Effect of Animated Phones as Facilitators in In-Person Group Discussion
by: Pan, Ziqi, et al.
Published: (2026)

ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents
by: Hoscilowicz, Jakub, et al.
Published: (2024)

UI-Evol: Automatic Knowledge Evolving for Computer Use Agents
by: Zhang, Ziyun, et al.
Published: (2025)

GUIrilla: A Scalable Framework for Automated Desktop UI Exploration
by: Garkot, Sofiya, et al.
Published: (2025)

LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation
by: Zhang, Li, et al.
Published: (2024)

CrowdGenUI: Aligning LLM-Based UI Generation with Crowdsourced User Preferences
by: Liu, Yimeng, et al.
Published: (2024)

Exploring the Role of Interaction Data to Empower End-User Decision-Making In UI Personalization
by: Alves, Sérgio, et al.
Published: (2026)

Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces
by: Lugoloobi, William, et al.
Published: (2026)

Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models
by: Parvez, Athar, et al.
Published: (2026)

VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
by: Song, Yunpeng, et al.
Published: (2023)

ReDemon UI: Reactive Synthesis by Demonstration for Web UI
by: Lee, Jay, et al.
Published: (2025)

The GenUI Study: Exploring the Design of Generative UI Tools to Support UX Practitioners and Beyond
by: Chen, Xiang 'Anthony', et al.
Published: (2025)

Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development
by: Haque, Sabrina, et al.
Published: (2025)

AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agents
by: Lu, Yuxuan, et al.
Published: (2025)

Beyond Automation: How UI/UX Designers Perceive AI as a Creative Partner in the Divergent Thinking Stages
by: Khan, Abidullah, et al.
Published: (2025)

SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI
by: Chen, Yunnong, et al.
Published: (2025)

User-Centric Design of UI for Mobile Banking Apps: Improving UI and Features for Better Customer Experience
by: Chitrakar, Luniva, et al.
Published: (2026)