AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917787199340544 |
|---|---|
| author | Huang, Wenhao Gu, Zhouhong Peng, Chenghao Li, Zhixu Liang, Jiaqing Xiao, Yanghua Wen, Liqian Chen, Zulong |
| author_facet | Huang, Wenhao Gu, Zhouhong Peng, Chenghao Li, Zhixu Liang, Jiaqing Xiao, Yanghua Wen, Liqian Chen, Zulong |
| contents | Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website, while language agents, empowered by large language models (LLMs), exhibit poor reusability in diverse web environments. In this work, we introduce the paradigm of generating web scrapers with LLMs and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently. AutoScraper leverages the hierarchical structure of HTML and similarity across different web pages for generating web scrapers. Besides, we propose a new executability metric for better measuring the performance of web scraper generation tasks. We conduct comprehensive experiments with multiple LLMs and demonstrate the effectiveness of our framework. Resources of this paper can be found at \url{https://github.com/EZ-hwh/AutoScraper} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_12753 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation Huang, Wenhao Gu, Zhouhong Peng, Chenghao Li, Zhixu Liang, Jiaqing Xiao, Yanghua Wen, Liqian Chen, Zulong Computation and Language Artificial Intelligence Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website, while language agents, empowered by large language models (LLMs), exhibit poor reusability in diverse web environments. In this work, we introduce the paradigm of generating web scrapers with LLMs and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently. AutoScraper leverages the hierarchical structure of HTML and similarity across different web pages for generating web scrapers. Besides, we propose a new executability metric for better measuring the performance of web scraper generation tasks. We conduct comprehensive experiments with multiple LLMs and demonstrate the effectiveness of our framework. Resources of this paper can be found at \url{https://github.com/EZ-hwh/AutoScraper} |
| title | AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.12753 |