Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.04759 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914743557554176 |
|---|---|
| author | Awobade, Busayo Oduwole, Mardiyyah Kolawole, Steven |
| author_facet | Awobade, Busayo Oduwole, Mardiyyah Kolawole, Steven |
| contents | Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04759 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models Awobade, Busayo Oduwole, Mardiyyah Kolawole, Steven Computation and Language Machine Learning Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models. |
| title | What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.04759 |