Unearthing Large Scale Domain-Specific Knowledge from Public Corpora
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918032383672320 |
|---|---|
| author | Fei, Zhaoye Shao, Yunfan Li, Linyang Zeng, Zhiyuan He, Conghui Guo, Qipeng Yan, Hang Lin, Dahua Qiu, Xipeng |
| author_facet | Fei, Zhaoye Shao, Yunfan Li, Linyang Zeng, Zhiyuan He, Conghui Guo, Qipeng Yan, Hang Lin, Dahua Qiu, Xipeng |
| contents | Large language models (LLMs) have demonstrated remarkable potential in various tasks, however, there remains a significant lack of open-source models and data for specific domains. Previous work has primarily focused on manually specifying resources and collecting high-quality data for specific domains, which is extremely time-consuming and labor-intensive. To address this limitation, we introduce large models into the data collection pipeline to guide the generation of domain-specific information and retrieve relevant data from Common Crawl (CC), a large public corpus. We refer to this approach as Retrieve-from-CC. It not only collects data related to domain-specific knowledge but also mines the data containing potential reasoning procedures from the public corpus. By applying this method, we have collected a knowledge domain-related dataset named Retrieve-Pile, which covers four main domains, including the sciences, humanities, and other categories. Through the analysis of , Retrieve-from-CC can effectively retrieve relevant data from the covered knowledge domains and significantly improve the performance in tests of mathematical and knowledge-related reasoning abilities. We have released Retrieve-Pile at https://huggingface.co/datasets/Query-of-CC/Retrieve-Pile. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_14624 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unearthing Large Scale Domain-Specific Knowledge from Public Corpora Fei, Zhaoye Shao, Yunfan Li, Linyang Zeng, Zhiyuan He, Conghui Guo, Qipeng Yan, Hang Lin, Dahua Qiu, Xipeng Computation and Language Large language models (LLMs) have demonstrated remarkable potential in various tasks, however, there remains a significant lack of open-source models and data for specific domains. Previous work has primarily focused on manually specifying resources and collecting high-quality data for specific domains, which is extremely time-consuming and labor-intensive. To address this limitation, we introduce large models into the data collection pipeline to guide the generation of domain-specific information and retrieve relevant data from Common Crawl (CC), a large public corpus. We refer to this approach as Retrieve-from-CC. It not only collects data related to domain-specific knowledge but also mines the data containing potential reasoning procedures from the public corpus. By applying this method, we have collected a knowledge domain-related dataset named Retrieve-Pile, which covers four main domains, including the sciences, humanities, and other categories. Through the analysis of , Retrieve-from-CC can effectively retrieve relevant data from the covered knowledge domains and significantly improve the performance in tests of mathematical and knowledge-related reasoning abilities. We have released Retrieve-Pile at https://huggingface.co/datasets/Query-of-CC/Retrieve-Pile. |
| title | Unearthing Large Scale Domain-Specific Knowledge from Public Corpora |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.14624 |