Efficiently Distilling LLMs for Edge Applications
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914738375491584 |
|---|---|
| author | Kundu, Achintya Lim, Fabian Chew, Aaron Wynter, Laura Chong, Penny Lee, Rhui Dih |
| author_facet | Kundu, Achintya Lim, Fabian Chew, Aaron Wynter, Laura Chong, Penny Lee, Rhui Dih |
| contents | Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01353 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficiently Distilling LLMs for Edge Applications Kundu, Achintya Lim, Fabian Chew, Aaron Wynter, Laura Chong, Penny Lee, Rhui Dih Machine Learning Artificial Intelligence Computation and Language Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time. |
| title | Efficiently Distilling LLMs for Edge Applications |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2404.01353 |