A Survey of Small Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916455831830528 |
|---|---|
| author | Van Nguyen, Chien Shen, Xuan Aponte, Ryan Xia, Yu Basu, Samyadeep Hu, Zhengmian Chen, Jian Parmar, Mihir Kunapuli, Sasidhar Barrow, Joe Wu, Junda Singh, Ashish Wang, Yu Gu, Jiuxiang Dernoncourt, Franck Ahmed, Nesreen K. Lipka, Nedim Zhang, Ruiyi Chen, Xiang Yu, Tong Kim, Sungchul Deilamsalehy, Hanieh Park, Namyong Rimer, Mike Zhang, Zhehao Yang, Huanrui Rossi, Ryan A. Nguyen, Thien Huu |
| author_facet | Van Nguyen, Chien Shen, Xuan Aponte, Ryan Xia, Yu Basu, Samyadeep Hu, Zhengmian Chen, Jian Parmar, Mihir Kunapuli, Sasidhar Barrow, Joe Wu, Junda Singh, Ashish Wang, Yu Gu, Jiuxiang Dernoncourt, Franck Ahmed, Nesreen K. Lipka, Nedim Zhang, Ruiyi Chen, Xiang Yu, Tong Kim, Sungchul Deilamsalehy, Hanieh Park, Namyong Rimer, Mike Zhang, Zhehao Yang, Huanrui Rossi, Ryan A. Nguyen, Thien Huu |
| contents | Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources, making them ideal for various settings including on-device, mobile, edge devices, among many others. In this article, we present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques. We propose a novel taxonomy for categorizing the methods used to optimize SLMs, including model compression, pruning, and quantization techniques. We summarize the benchmark datasets that are useful for benchmarking SLMs along with the evaluation metrics commonly used. Additionally, we highlight key open challenges that remain to be addressed. Our survey aims to serve as a valuable resource for researchers and practitioners interested in developing and deploying small yet efficient language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_20011 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Survey of Small Language Models Van Nguyen, Chien Shen, Xuan Aponte, Ryan Xia, Yu Basu, Samyadeep Hu, Zhengmian Chen, Jian Parmar, Mihir Kunapuli, Sasidhar Barrow, Joe Wu, Junda Singh, Ashish Wang, Yu Gu, Jiuxiang Dernoncourt, Franck Ahmed, Nesreen K. Lipka, Nedim Zhang, Ruiyi Chen, Xiang Yu, Tong Kim, Sungchul Deilamsalehy, Hanieh Park, Namyong Rimer, Mike Zhang, Zhehao Yang, Huanrui Rossi, Ryan A. Nguyen, Thien Huu Computation and Language Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources, making them ideal for various settings including on-device, mobile, edge devices, among many others. In this article, we present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques. We propose a novel taxonomy for categorizing the methods used to optimize SLMs, including model compression, pruning, and quantization techniques. We summarize the benchmark datasets that are useful for benchmarking SLMs along with the evaluation metrics commonly used. Additionally, we highlight key open challenges that remain to be addressed. Our survey aims to serve as a valuable resource for researchers and practitioners interested in developing and deploying small yet efficient language models. |
| title | A Survey of Small Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.20011 |