Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.05555 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910733038518272 |
|---|---|
| author | Chen, Yunnong Xiao, Shuhong Li, Jiazhi Zhou, Tingting Chang, Yanfang Zhen, Yankun Sun, Lingyun Chen, Liuqing |
| author_facet | Chen, Yunnong Xiao, Shuhong Li, Jiazhi Zhou, Tingting Chang, Yanfang Zhen, Yankun Sun, Lingyun Chen, Liuqing |
| contents | Automatically constructing GUI groups of different granularities constitutes a critical intelligent step towards automating GUI design and implementation tasks. Specifically, in the industrial GUI-to-code process, fragmented layers may decrease the readability and maintainability of generated code, which can be alleviated by grouping semantically consistent fragmented layers in the design prototypes. This study aims to propose a graph-learning-based approach to tackle the fragmented layer grouping problem according to multi-modal information in design prototypes. Our graph learning module consists of self-attention and graph neural network modules. By taking the multimodal fused representation of GUI layers as input, we innovatively group fragmented layers by classifying GUI layers and regressing the bounding boxes of the corresponding GUI components simultaneously. Experiments on two real-world datasets demonstrate that our model achieves state-of-the-art performance. A further user study is also conducted to validate that our approach can assist an intelligent downstream tool in generating more maintainable and readable front-end code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_05555 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fragmented Layer Grouping in GUI Designs Through Graph Learning Based on Multimodal Information Chen, Yunnong Xiao, Shuhong Li, Jiazhi Zhou, Tingting Chang, Yanfang Zhen, Yankun Sun, Lingyun Chen, Liuqing Software Engineering Artificial Intelligence Automatically constructing GUI groups of different granularities constitutes a critical intelligent step towards automating GUI design and implementation tasks. Specifically, in the industrial GUI-to-code process, fragmented layers may decrease the readability and maintainability of generated code, which can be alleviated by grouping semantically consistent fragmented layers in the design prototypes. This study aims to propose a graph-learning-based approach to tackle the fragmented layer grouping problem according to multi-modal information in design prototypes. Our graph learning module consists of self-attention and graph neural network modules. By taking the multimodal fused representation of GUI layers as input, we innovatively group fragmented layers by classifying GUI layers and regressing the bounding boxes of the corresponding GUI components simultaneously. Experiments on two real-world datasets demonstrate that our model achieves state-of-the-art performance. A further user study is also conducted to validate that our approach can assist an intelligent downstream tool in generating more maintainable and readable front-end code. |
| title | Fragmented Layer Grouping in GUI Designs Through Graph Learning Based on Multimodal Information |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2412.05555 |