Improving User Interface Generation Models from Designer Feedback
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912908737249280 |
|---|---|
| author | Wu, Jason Swearngin, Amanda Vajjala, Arun Krishna Leung, Alan Nichols, Jeffrey Barik, Titus |
| author_facet | Wu, Jason Swearngin, Amanda Vajjala, Arun Krishna Leung, Alan Nichols, Jeffrey Barik, Titus |
| contents | Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore the rich rationale used to critique and improve UI designs. In this paper, we investigate several approaches for designers to give feedback to UI generation models, using familiar interactions such as commenting, sketching and direct manipulation. We first perform an evaluation with 21 designers where they gave feedback using these interactions, which resulted in 1500 design annotations. We then use this data to finetune a series of LLMs to generate higher quality UIs. Finally, we evaluate these models with human judges, and we find that our designer-aligned approaches outperform models trained with traditional ranking feedback and all tested baselines, including GPT-5. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16779 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving User Interface Generation Models from Designer Feedback Wu, Jason Swearngin, Amanda Vajjala, Arun Krishna Leung, Alan Nichols, Jeffrey Barik, Titus Human-Computer Interaction Machine Learning Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore the rich rationale used to critique and improve UI designs. In this paper, we investigate several approaches for designers to give feedback to UI generation models, using familiar interactions such as commenting, sketching and direct manipulation. We first perform an evaluation with 21 designers where they gave feedback using these interactions, which resulted in 1500 design annotations. We then use this data to finetune a series of LLMs to generate higher quality UIs. Finally, we evaluate these models with human judges, and we find that our designer-aligned approaches outperform models trained with traditional ranking feedback and all tested baselines, including GPT-5. |
| title | Improving User Interface Generation Models from Designer Feedback |
| topic | Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2509.16779 |