Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915115559813120 |
|---|---|
| author | Cruz, Jan Christian Blaise Aji, Alham Fikri |
| author_facet | Cruz, Jan Christian Blaise Aji, Alham Fikri |
| contents | In this paper, we propose the use of simple knowledge distillation to produce smaller and more efficient single-language transformers from Massively Multilingual Transformers (MMTs) to alleviate tradeoffs associated with the use of such in low-resource settings. Using Tagalog as a case study, we show that these smaller single-language models perform on-par with strong baselines in a variety of benchmark tasks in a much more efficient manner. Furthermore, we investigate additional steps during the distillation process that improves the soft-supervision of the target language, and provide a number of analyses and ablations to show the efficacy of the proposed method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_12660 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation Cruz, Jan Christian Blaise Aji, Alham Fikri Computation and Language In this paper, we propose the use of simple knowledge distillation to produce smaller and more efficient single-language transformers from Massively Multilingual Transformers (MMTs) to alleviate tradeoffs associated with the use of such in low-resource settings. Using Tagalog as a case study, we show that these smaller single-language models perform on-par with strong baselines in a variety of benchmark tasks in a much more efficient manner. Furthermore, we investigate additional steps during the distillation process that improves the soft-supervision of the target language, and provide a number of analyses and ablations to show the efficacy of the proposed method. |
| title | Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.12660 |