Text Quality-Based Pruning for Efficient Training of Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909197826785280 |
|---|---|
| author | Sharma, Vasu Padthe, Karthik Ardalani, Newsha Tirumala, Kushal Howes, Russell Xu, Hu Huang, Po-Yao Li, Shang-Wen Aghajanyan, Armen Ghosh, Gargi Zettlemoyer, Luke |
| author_facet | Sharma, Vasu Padthe, Karthik Ardalani, Newsha Tirumala, Kushal Howes, Russell Xu, Hu Huang, Po-Yao Li, Shang-Wen Aghajanyan, Armen Ghosh, Gargi Zettlemoyer, Luke |
| contents | In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious. In this paper we propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets in a model agnostic manner to assign the text instances a "quality score".
By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances, leading to improved training efficiency for LM models. Experimental results over multiple models and datasets demonstrate the efficacy of this approach, showcasing substantial gains in training effectiveness and highlighting the potential for resource-efficient LM training.
For example, we observe an absolute accuracy improvement of 0.9% averaged over 14 downstream evaluation tasks for multiple LM models while using 40% lesser data and training 42% faster when training on the OpenWebText dataset and 0.8% average absolute accuracy improvement while using 20% lesser data and training 21% faster on the Wikipedia dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01582 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Text Quality-Based Pruning for Efficient Training of Language Models Sharma, Vasu Padthe, Karthik Ardalani, Newsha Tirumala, Kushal Howes, Russell Xu, Hu Huang, Po-Yao Li, Shang-Wen Aghajanyan, Armen Ghosh, Gargi Zettlemoyer, Luke Computation and Language Artificial Intelligence Machine Learning In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious. In this paper we propose a novel method for numerically evaluating text quality in large unlabelled NLP datasets in a model agnostic manner to assign the text instances a "quality score". By proposing the text quality metric, the paper establishes a framework to identify and eliminate low-quality text instances, leading to improved training efficiency for LM models. Experimental results over multiple models and datasets demonstrate the efficacy of this approach, showcasing substantial gains in training effectiveness and highlighting the potential for resource-efficient LM training. For example, we observe an absolute accuracy improvement of 0.9% averaged over 14 downstream evaluation tasks for multiple LM models while using 40% lesser data and training 42% faster when training on the OpenWebText dataset and 0.8% average absolute accuracy improvement while using 20% lesser data and training 21% faster on the Wikipedia dataset. |
| title | Text Quality-Based Pruning for Efficient Training of Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2405.01582 |