Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917362723192832 |
|---|---|
| author | Zhang, Houston H. Zhang, Tao Lin, Baoze Xue, Yuanqi Zhu, Yincheng Liu, Huan Gu, Li Ye, Linfeng Wang, Ziqiang Zuo, Xinxin Wang, Yang Yu, Yuanhao Chi, Zhixiang |
| author_facet | Zhang, Houston H. Zhang, Tao Lin, Baoze Xue, Yuanqi Zhu, Yincheng Liu, Huan Gu, Li Ye, Linfeng Wang, Ziqiang Zuo, Xinxin Wang, Yang Yu, Yuanhao Chi, Zhixiang |
| contents | User interface to code (UI2Code) aims to generate executable code that can faithfully reconstruct a given input UI. Prior work focuses largely on web pages and mobile screens, leaving app widgets underexplored. Unlike web or mobile UIs with rich hierarchical context, widgets are compact, context-free micro-interfaces that summarize key information through dense layouts and iconography under strict spatial constraints. Moreover, while (image, code) pairs are widely available for web or mobile UIs, widget designs are proprietary and lack accessible markup. We formalize this setting as the Widget-to-Code (Widget2Code) and introduce an image-only widget benchmark with fine-grained, multi-dimensional evaluation metrics. Benchmarking shows that although generalized multimodal large language models (MLLMs) outperform specialized UI2Code methods, they still produce unreliable and visually inconsistent code. To address these limitations, we develop a baseline that jointly advances perceptual understanding and structured code generation. At the perceptual level, we follow widget design principles to assemble atomic components into complete layouts, equipped with icon retrieval and reusable visualization modules. At the system level, we design an end-to-end infrastructure, WidgetFactory, which includes a framework-agnostic widget-tailored domain-specific language (WidgetDSL) and a compiler that translates it into multiple front-end implementations (e.g., React, HTML/CSS). An adaptive rendering module further refines spatial dimensions to satisfy compactness constraints. Together, these contributions substantially enhance visual fidelity, establishing a strong baseline and unified infrastructure for future Widget2Code research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_19918 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs Zhang, Houston H. Zhang, Tao Lin, Baoze Xue, Yuanqi Zhu, Yincheng Liu, Huan Gu, Li Ye, Linfeng Wang, Ziqiang Zuo, Xinxin Wang, Yang Yu, Yuanhao Chi, Zhixiang Computer Vision and Pattern Recognition User interface to code (UI2Code) aims to generate executable code that can faithfully reconstruct a given input UI. Prior work focuses largely on web pages and mobile screens, leaving app widgets underexplored. Unlike web or mobile UIs with rich hierarchical context, widgets are compact, context-free micro-interfaces that summarize key information through dense layouts and iconography under strict spatial constraints. Moreover, while (image, code) pairs are widely available for web or mobile UIs, widget designs are proprietary and lack accessible markup. We formalize this setting as the Widget-to-Code (Widget2Code) and introduce an image-only widget benchmark with fine-grained, multi-dimensional evaluation metrics. Benchmarking shows that although generalized multimodal large language models (MLLMs) outperform specialized UI2Code methods, they still produce unreliable and visually inconsistent code. To address these limitations, we develop a baseline that jointly advances perceptual understanding and structured code generation. At the perceptual level, we follow widget design principles to assemble atomic components into complete layouts, equipped with icon retrieval and reusable visualization modules. At the system level, we design an end-to-end infrastructure, WidgetFactory, which includes a framework-agnostic widget-tailored domain-specific language (WidgetDSL) and a compiler that translates it into multiple front-end implementations (e.g., React, HTML/CSS). An adaptive rendering module further refines spatial dimensions to satisfy compactness constraints. Together, these contributions substantially enhance visual fidelity, establishing a strong baseline and unified infrastructure for future Widget2Code research. |
| title | Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.19918 |