Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.08850 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909286461865984 |
|---|---|
| author | Duan, Peitong Chen, Chin-yi Li, Gang Hartmann, Bjoern Li, Yang |
| author_facet | Duan, Peitong Chen, Chin-yi Li, Gang Hartmann, Bjoern Li, Yang |
| contents | Automated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation, in particular, holds the promise of generalizability to a wide variety of UI types and evaluation tasks. However, current LLM-based techniques do not yet match the performance of human evaluators. We hypothesize that automatic evaluation can be improved by collecting a targeted UI feedback dataset and then using this dataset to enhance the performance of general-purpose LLMs. We present a targeted dataset of 3,059 design critiques and quality ratings for 983 mobile UIs, collected from seven experienced designers. We carried out an in-depth analysis to characterize the dataset's features. We then applied this dataset to achieve a 55% performance gain in LLM-generated UI feedback via various few-shot and visual prompting techniques. We also discuss future applications of this dataset, including training a reward model for generative UI techniques, and fine-tuning a tool-agnostic multi-modal LLM that automates UI evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08850 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset Duan, Peitong Chen, Chin-yi Li, Gang Hartmann, Bjoern Li, Yang Human-Computer Interaction Artificial Intelligence Automated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation, in particular, holds the promise of generalizability to a wide variety of UI types and evaluation tasks. However, current LLM-based techniques do not yet match the performance of human evaluators. We hypothesize that automatic evaluation can be improved by collecting a targeted UI feedback dataset and then using this dataset to enhance the performance of general-purpose LLMs. We present a targeted dataset of 3,059 design critiques and quality ratings for 983 mobile UIs, collected from seven experienced designers. We carried out an in-depth analysis to characterize the dataset's features. We then applied this dataset to achieve a 55% performance gain in LLM-generated UI feedback via various few-shot and visual prompting techniques. We also discuss future applications of this dataset, including training a reward model for generative UI techniques, and fine-tuning a tool-agnostic multi-modal LLM that automates UI evaluation. |
| title | UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2407.08850 |