Cover Image

Context-Aware Workflow Decomposition for Automated Mobile UI Annotation Using Multimodal Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Parvez, Athar, Mufti, Muhammad Jawad, Gull, Muqaddas, Hammad, Omar
Format:	Preprint
Published:	2026
Subjects:	Human-Computer Interaction
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MUIAnno: An Expert-Annotated Dataset and Evaluation Benchmark for Mobile UI Understanding
by: Parvez, Athar, et al.
Published: (2026)

VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
by: Song, Yunpeng, et al.
Published: (2023)

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
by: You, Keen, et al.
Published: (2024)

Generating Automatic Feedback on UI Mockups with Large Language Models
by: Duan, Peitong, et al.
Published: (2024)

GhostUI: Unveiling Hidden Interactions in Mobile UI
by: Kweon, Minkyu, et al.
Published: (2026)

Agent-Initiated Interaction in Phone UI Automation
by: Kahlon, Noam, et al.
Published: (2025)

User-Centric Design of UI for Mobile Banking Apps: Improving UI and Features for Better Customer Experience
by: Chitrakar, Luniva, et al.
Published: (2026)

LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation
by: Zhang, Li, et al.
Published: (2024)

MultiSurf-GPT: Facilitating Context-Aware Reasoning with Large-Scale Language Models for Multimodal Surface Sensing
by: Hu, Yongquan, et al.
Published: (2024)

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling
by: Feng, Sidong, et al.
Published: (2024)

IDA: Breaking Barriers in No-code UI Automation Through Large Language Models and Human-Centric Design
by: Shlomov, Segev, et al.
Published: (2024)

Automating UI Optimization through Multi-Agentic Reasoning
by: Li, Zhipeng, et al.
Published: (2026)

Automated UI Interface Generation via Diffusion Models: Enhancing Personalization and Efficiency
by: Duan, Yifei, et al.
Published: (2025)

Towards Context-Aware Modeling of Situation Awareness in Conditionally Automated Driving
by: Avetisyan, Lilit, et al.
Published: (2024)

Do Large Language Models Speak Scientific Workflows?
by: Yildiz, Orcun, et al.
Published: (2024)

Macaron-A2UI: A Model for Generative UI in Personal Agents
by: Kong, Fancy, et al.
Published: (2026)

MLLM-Based UI2Code Automation Guided by UI Layout Information
by: Wu, Fan, et al.
Published: (2025)

Affordances of Sketched Notations for Multimodal UI Design and Development Tools
by: Ross, Sam H., et al.
Published: (2025)

Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts
by: Huang, Tian, et al.
Published: (2024)

Generalizable Diabetes Risk Stratification via Hybrid Machine Learning Models
by: Parvez, Athar, et al.
Published: (2025)

AutoGameUI: Constructing High-Fidelity GameUI via Multimodal Correspondence Matching
by: Tang, Zhongliang, et al.
Published: (2024)

Inferring Alt-text For UI Icons With Large Language Models During App Development
by: Haque, Sabrina, et al.
Published: (2024)

Privacy Starts with UI: Privacy Patterns and Designer Perspectives in UI/UX Practice
by: Maloku, Anxhela, et al.
Published: (2026)

UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing
by: Wang, Junling, et al.
Published: (2026)

Leveraging Large Language Models to Enhance Domain Expert Inclusion in Data Science Workflows
by: Shih, Jasmine Y., et al.
Published: (2024)

CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only
by: Cho, Junhee, et al.
Published: (2024)

MapQaTor: An Extensible Framework for Efficient Annotation of Map-Based QA Datasets
by: Dihan, Mahir Labib, et al.
Published: (2024)

CrowdGenUI: Aligning LLM-Based UI Generation with Crowdsourced User Preferences
by: Liu, Yimeng, et al.
Published: (2024)

SVG Decomposition for Enhancing Large Multimodal Models Visualization Comprehension: A Study with Floor Plans
by: Lee, Jeongah, et al.
Published: (2025)

Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support
by: Xu, Haowen, et al.
Published: (2025)

Towards Enhanced Context Awareness with Vision-based Multimodal Interfaces
by: Hu, Yongquan, et al.
Published: (2024)

AttenTrack: Mobile User Attention Awareness Based on Context and External Distractions
by: Lin, Yutong, et al.
Published: (2025)

LLM-for-X: Application-agnostic Integration of Large Language Models to Support Personal Writing Workflows
by: Teufelberger, Lukas, et al.
Published: (2024)

Supporting Business Document Workflows via Collection-Centric Information Foraging with Large Language Models
by: Fok, Raymond, et al.
Published: (2024)

From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts
by: Zhang, Zhuohao Jerry, et al.
Published: (2024)

DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces
by: Xu, Yuan, et al.
Published: (2025)

The GenUI Study: Exploring the Design of Generative UI Tools to Support UX Practitioners and Beyond
by: Chen, Xiang 'Anthony', et al.
Published: (2025)

SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM Reasoning
by: Li, Zhipeng, et al.
Published: (2024)

Optimizing Gesture Recognition for Seamless UI Interaction Using Convolutional Neural Networks
by: Sun, Qi, et al.
Published: (2024)

Beyond Automation: How UI/UX Designers Perceive AI as a Creative Partner in the Divergent Thinking Stages
by: Khan, Abidullah, et al.
Published: (2025)