Cover Image

From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Zhuohao Jerry, Schoop, Eldon, Nichols, Jeffrey, Mahajan, Anuj, Swearngin, Amanda
Format:	Preprint
Published:	2024
Subjects:	Human-Computer Interaction
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
by: You, Keen, et al.
Published: (2024)

Misty: UI Prototyping Through Interactive Conceptual Blending
by: Lu, Yuwen, et al.
Published: (2024)

AXNav: Replaying Accessibility Tests from Natural Language
by: Taeb, Maryam, et al.
Published: (2023)

Mapping the Design Space of User Experience for Computer Use Agents
by: Cheng, Ruijia, et al.
Published: (2026)

Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM
by: Beason, Jazbo, et al.
Published: (2025)

AgentBuilder: Exploring Scaffolds for Prototyping User Experiences of Interface Agents
by: Liang, Jenny T., et al.
Published: (2025)

FlowEval: Reference-based Evaluation of Generated User Interfaces
by: Wu, Jason, et al.
Published: (2026)

Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis
by: Schoop, Eldon, et al.
Published: (2022)

UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
by: Wu, Jason, et al.
Published: (2024)

Automated Code Fix Suggestions for Accessibility Issues in Mobile Apps
by: Mehralian, Forough, et al.
Published: (2024)

Improving User Interface Generation Models from Designer Feedback
by: Wu, Jason, et al.
Published: (2025)

MUIAnno: An Expert-Annotated Dataset and Evaluation Benchmark for Mobile UI Understanding
by: Parvez, Athar, et al.
Published: (2026)

GhostUI: Unveiling Hidden Interactions in Mobile UI
by: Kweon, Minkyu, et al.
Published: (2026)

The Way We Notice, That's What Really Matters: Instantiating UI Components with Distinguishing Variations
by: Vaithilingam, Priyan, et al.
Published: (2026)

UIClip: A Data-driven Model for Assessing User Interface Design
by: Wu, Jason, et al.
Published: (2024)

UI Remix: Supporting UI Design Through Interactive Example Retrieval and Remixing
by: Wang, Junling, et al.
Published: (2026)

Agent-Initiated Interaction in Phone UI Automation
by: Kahlon, Noam, et al.
Published: (2025)

SlideAudit: A Dataset and Taxonomy for Automated Evaluation of Presentation Slides
by: Zhang, Zhuohao Jerry, et al.
Published: (2025)

Open WebUI: An Open, Extensible, and Usable Interface for AI Interaction
by: Baek, Jaeryang, et al.
Published: (2025)

VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning
by: Song, Yunpeng, et al.
Published: (2023)

User-Centric Design of UI for Mobile Banking Apps: Improving UI and Features for Better Customer Experience
by: Chitrakar, Luniva, et al.
Published: (2026)

Macaron-A2UI: A Model for Generative UI in Personal Agents
by: Kong, Fancy, et al.
Published: (2026)

Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations
by: Liang, Chen, et al.
Published: (2026)

From Interaction to Attitude: Exploring the Impact of Human-AI Cooperation on Mental Illness Stigma
by: Song, Tianqi, et al.
Published: (2025)

Evaluating the Impact of Community Oversight for Managing Mobile Privacy and Security
by: Akter, Mamtaj, et al.
Published: (2023)

OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering
by: Li, Jiahao Nick, et al.
Published: (2024)

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling
by: Feng, Sidong, et al.
Published: (2024)

SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI
by: Chen, Yunnong, et al.
Published: (2025)

Towards Social AI: A Survey on Understanding Social Interactions
by: Lee, Sangmin, et al.
Published: (2024)

Towards Safer and Understandable Driver Intention Prediction
by: Karuppasamy, Mukilan, et al.
Published: (2025)

Designing Safe and Engaging AI Experiences for Children: Towards the Definition of Best Practices in UI/UX Design
by: Ragone, Grazia, et al.
Published: (2024)

Towards Understanding the Impact of Guidance in Data Visualization Systems for Domain Experts
by: Qiu, Sherry, et al.
Published: (2024)

RWKV-UI: UI Understanding with Enhanced Perception and Reasoning
by: Yang, Jiaxi, et al.
Published: (2025)

Reframing Human-Robot Interaction Through Extended Reality: Unlocking Safer, Smarter, and More Empathic Interactions with Virtual Robots and Foundation Models
by: Zhang, Yuchong, et al.
Published: (2025)

MAIC-UI: Making Interactive Courseware with Generative UI
by: Tu, Shangqing, et al.
Published: (2026)

UFO: A UI-Focused Agent for Windows OS Interaction
by: Zhang, Chaoyun, et al.
Published: (2024)

ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agent
by: Zhao, Yuheng, et al.
Published: (2025)

Effective and Scalable Math Support: Evidence on the Impact of an AI- Tutor on Math Achievement in Ghana
by: Henkel, Owen, et al.
Published: (2024)

Provocation on Expertise in Social Impact Evaluations of Generative AI (and Beyond)
by: Kahn, Zoe, et al.
Published: (2024)

Optimizing Gesture Recognition for Seamless UI Interaction Using Convolutional Neural Networks
by: Sun, Qi, et al.
Published: (2024)