Cover Image

How Smart Is Your GUI Agent? A Framework for the Future of Software Interaction

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Feng, Sidong, Chen, Chunyang
Format:	Preprint
Published:	2026
Subjects:	Software Engineering Artificial Intelligence Computer Vision and Pattern Recognition Human-Computer Interaction
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

GUI Agents for Continual Game Generation
by: Huang, Yixu, et al.
Published: (2026)

MAIA: A Collaborative Medical AI Platform for Integrated Healthcare Innovation
by: Bendazzoli, Simone, et al.
Published: (2025)

Automating Crash Diagram Generation Using Vision-Language Models: A Case Study on Multi-Lane Roundabouts
by: Lu, Xiao, et al.
Published: (2026)

Explain and Monitor Deep Learning Models for Computer Vision using Obz AI
by: Chung, Neo Christopher, et al.
Published: (2025)

How Developers Interact with AI: A Taxonomy of Human-AI Collaboration in Software Engineering
by: Treude, Christoph, et al.
Published: (2025)

ML-EcoLyzer: Quantifying the Environmental Cost of Machine Learning Inference Across Frameworks and Hardware
by: Minoza, Jose Marie Antonio, et al.
Published: (2025)

AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild
by: Sun, Jiazheng, et al.
Published: (2026)

Qualitative Evaluation of LLM-Designed GUI
by: Sawicki, Bartosz, et al.
Published: (2026)

From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering
by: Dong, Tao, et al.
Published: (2025)

Real-Time Drivers' Drowsiness Detection and Analysis through Deep Learning
by: Zaman, ANK, et al.
Published: (2025)

UI-TARS: Pioneering Automated GUI Interaction with Native Agents
by: Qin, Yujia, et al.
Published: (2025)

Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
by: Liang, Jenny T., et al.
Published: (2024)

Interacting with AI Reasoning Models: Harnessing "Thoughts" for AI-Driven Software Engineering
by: Treude, Christoph, et al.
Published: (2025)

Adoption of Generative Artificial Intelligence in the German Software Engineering Industry: An Empirical Study
by: Felder, Ludwig, et al.
Published: (2026)

Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging
by: Liu, Zhilin, et al.
Published: (2026)

How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions
by: Tang, Ningzhi, et al.
Published: (2026)

Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025
by: Huang, Ruanqianqian, et al.
Published: (2025)

Explorer: Robust Collection of Interactable GUI Elements
by: Chaimalas, Iason, et al.
Published: (2025)

Towards an Appropriate Level of Reliance on AI: A Preliminary Reliance-Control Framework for AI in Software Engineering
by: Ferino, Samuel, et al.
Published: (2026)

From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering
by: Feng, Dana, et al.
Published: (2026)

MP-GUI: Modality Perception with MLLMs for GUI Understanding
by: Wang, Ziwei, et al.
Published: (2025)

Real-Time Multimodal Data Collection Using Smartwatches and Its Visualization in Education
by: Becerra, Alvaro, et al.
Published: (2025)

MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs
by: Krishnavajjala, Arun, et al.
Published: (2024)

An Empirical Study of Bugs in Data Visualization Libraries
by: Lu, Weiqi, et al.
Published: (2025)

History-Aware Reasoning for GUI Agents
by: Wang, Ziwei, et al.
Published: (2025)

Human-In-the-Loop Software Development Agents
by: Takerngsaksiri, Wannita, et al.
Published: (2024)

A Survey on (M)LLM-Based GUI Agents
by: Tang, Fei, et al.
Published: (2025)

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks
by: Bai, Longju, et al.
Published: (2026)

Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraph
by: Exarhakos, Vassilios, et al.
Published: (2026)

The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study
by: Mohamed, Amr, et al.
Published: (2025)

Single Conversation Methodology: A Human-Centered Protocol for AI-Assisted Software Development
by: Escobedo, Salvador D.
Published: (2025)

Thoughtful Things: Building Human-Centric Smart Devices with Small Language Models
by: King, Evan, et al.
Published: (2024)

Software Vulnerability Management in the Era of Artificial Intelligence: An Industry Perspective
by: Kholoosi, M. Mehdi, et al.
Published: (2025)

Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping
by: Xiao, Jingyu, et al.
Published: (2024)

DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
by: Wu, Hang, et al.
Published: (2025)

LLMs' Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters
by: Tabarsi, Benyamin, et al.
Published: (2025)

A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development
by: Geyer, Werner, et al.
Published: (2025)

Prompt-with-Me: in-IDE Structured Prompt Management for LLM-Driven Software Engineering
by: Li, Ziyou, et al.
Published: (2025)

CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing
by: Hu, Haobo, et al.
Published: (2026)

How Do Hackathons Foster Creativity? Towards AI Collaborative Evaluation of Creativity at Scale
by: Falk, Jeanette, et al.
Published: (2025)