Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909136639229952 |
|---|---|
| author | Laurençon, Hugo Tronchon, Léo Sanh, Victor |
| author_facet | Laurençon, Hugo Tronchon, Léo Sanh, Victor |
| contents | Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_09029 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset Laurençon, Hugo Tronchon, Léo Sanh, Victor Human-Computer Interaction Artificial Intelligence Computer Vision and Pattern Recognition Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight. |
| title | Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset |
| topic | Human-Computer Interaction Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.09029 |