Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2409.14705 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929509893144576 |
|---|---|
| author | Chang, Ernie Lin, Pin-Jie Li, Yang Zhao, Changsheng Kim, Daeil Rabatin, Rastislav Liu, Zechun Shi, Yangyang Chandra, Vikas |
| author_facet | Chang, Ernie Lin, Pin-Jie Li, Yang Zhao, Changsheng Kim, Daeil Rabatin, Rastislav Liu, Zechun Shi, Yangyang Chandra, Vikas |
| contents | Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows to select large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance while preserving its effectiveness on other tasks. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with $\sim$1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_14705 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Target-Aware Language Modeling via Granular Data Sampling Chang, Ernie Lin, Pin-Jie Li, Yang Zhao, Changsheng Kim, Daeil Rabatin, Rastislav Liu, Zechun Shi, Yangyang Chandra, Vikas Computation and Language Artificial Intelligence Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows to select large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance while preserving its effectiveness on other tasks. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with $\sim$1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B. |
| title | Target-Aware Language Modeling via Granular Data Sampling |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.14705 |