Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.02851 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915417975422976 |
|---|---|
| author | Zhang, Franklin Zhang, Sonya Halevy, Alon |
| author_facet | Zhang, Franklin Zhang, Sonya Halevy, Alon |
| contents | Constructing specialized content corpora from vast, unstructured web sources for domain-specific applications poses substantial data curation challenges. In this paper, we introduce a streamlined approach for generating high-quality, domain-specific corpora by efficiently acquiring, filtering, structuring, and cleaning web-based data. We showcase how Large Language Models (LLMs) can be leveraged to address complex data curation at scale, and propose a strategical framework incorporating LLM-enhanced techniques for structured content extraction and semantic deduplication. We validate our approach in the behavior education domain through its integration into 30 Day Me, a habit formation application. Our data pipeline, named 30DayGen, enabled the extraction and synthesis of 3,531 unique 30-day challenges from over 15K webpages. A user survey reports a satisfaction score of 4.3 out of 5, with 91% of respondents indicating willingness to use the curated content for their habit-formation goals. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_02851 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Leveraging LLMs to Create Content Corpora for Niche Domains Zhang, Franklin Zhang, Sonya Halevy, Alon Computation and Language Artificial Intelligence Computers and Society I.2.7; H.3.1; H.3.3 Constructing specialized content corpora from vast, unstructured web sources for domain-specific applications poses substantial data curation challenges. In this paper, we introduce a streamlined approach for generating high-quality, domain-specific corpora by efficiently acquiring, filtering, structuring, and cleaning web-based data. We showcase how Large Language Models (LLMs) can be leveraged to address complex data curation at scale, and propose a strategical framework incorporating LLM-enhanced techniques for structured content extraction and semantic deduplication. We validate our approach in the behavior education domain through its integration into 30 Day Me, a habit formation application. Our data pipeline, named 30DayGen, enabled the extraction and synthesis of 3,531 unique 30-day challenges from over 15K webpages. A user survey reports a satisfaction score of 4.3 out of 5, with 91% of respondents indicating willingness to use the curated content for their habit-formation goals. |
| title | Leveraging LLMs to Create Content Corpora for Niche Domains |
| topic | Computation and Language Artificial Intelligence Computers and Society I.2.7; H.3.1; H.3.3 |
| url | https://arxiv.org/abs/2505.02851 |