PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909380327243776 |
|---|---|
| author | Yi, Rongjie Li, Xiang Xie, Weikai Lu, Zhenyan Wang, Chenghua Zhou, Ao Wang, Shangguang Zhang, Xiwen Xu, Mengwei |
| author_facet | Yi, Rongjie Li, Xiang Xie, Weikai Lu, Zhenyan Wang, Chenghua Zhou, Ao Wang, Shangguang Zhang, Xiwen Xu, Mengwei |
| contents | The interest in developing small language models (SLM) for on-device deployment is fast growing. However, the existing SLM design hardly considers the device hardware characteristics. Instead, this work presents a simple yet effective principle for SLM design: architecture searching for (near-)optimal runtime efficiency before pre-training. Guided by this principle, we develop PhoneLM SLM family (currently with 0.5B and 1.5B versions), that acheive the state-of-the-art capability-efficiency tradeoff among those with similar parameter size. We fully open-source the code, weights, and training datasets of PhoneLM for reproducibility and transparency, including both base and instructed versions. We also release a finetuned version of PhoneLM capable of accurate Android Intent invocation, and an end-to-end Android demo. All materials are available at https://github.com/UbiquitousLearning/PhoneLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05046 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training Yi, Rongjie Li, Xiang Xie, Weikai Lu, Zhenyan Wang, Chenghua Zhou, Ao Wang, Shangguang Zhang, Xiwen Xu, Mengwei Computation and Language Artificial Intelligence Machine Learning The interest in developing small language models (SLM) for on-device deployment is fast growing. However, the existing SLM design hardly considers the device hardware characteristics. Instead, this work presents a simple yet effective principle for SLM design: architecture searching for (near-)optimal runtime efficiency before pre-training. Guided by this principle, we develop PhoneLM SLM family (currently with 0.5B and 1.5B versions), that acheive the state-of-the-art capability-efficiency tradeoff among those with similar parameter size. We fully open-source the code, weights, and training datasets of PhoneLM for reproducibility and transparency, including both base and instructed versions. We also release a finetuned version of PhoneLM capable of accurate Android Intent invocation, and an end-to-end Android demo. All materials are available at https://github.com/UbiquitousLearning/PhoneLM. |
| title | PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.05046 |