PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912283246985216 |
|---|---|
| author | Deng, Cheng Sun, Luoyang Jiang, Jiwen Zeng, Yongcheng Wu, Xinjian Zhao, Wenxin Xiao, Qingfa Wang, Jiachuan Li, Haoyang Chen, Lei Ni, Lionel M. Zhang, Haifeng Wang, Jun |
| author_facet | Deng, Cheng Sun, Luoyang Jiang, Jiwen Zeng, Yongcheng Wu, Xinjian Zhao, Wenxin Xiao, Qingfa Wang, Jiachuan Li, Haoyang Chen, Lei Ni, Lionel M. Zhang, Haifeng Wang, Jun |
| contents | While scaling laws have been continuously validated in large language models (LLMs) with increasing model parameters, the inherent tension between the inference demands of LLMs and the limited resources of edge devices poses a critical challenge to the development of edge intelligence. Recently, numerous small language models have emerged, aiming to distill the capabilities of LLMs into smaller footprints. However, these models often retain the fundamental architectural principles of their larger counterparts, still imposing considerable strain on the storage and bandwidth capacities of edge devices. In this paper, we introduce the PLM, a Peripheral Language Model, developed through a co-design process that jointly optimizes model architecture and edge system constraints. The PLM utilizes a Multi-head Latent Attention mechanism and employs the squared ReLU activation function to encourage sparsity, thereby reducing peak memory footprint during inference. During training, we collect and reorganize open-source datasets, implement a multi-phase training strategy, and empirically investigate the Warmup-Stable-Decay-Constant (WSDC) learning rate scheduler. Additionally, we incorporate Reinforcement Learning from Human Feedback (RLHF) by adopting the ARIES preference learning approach. Following a two-phase SFT process, this method yields performance gains of 2% in general tasks, 9% in the GSM8K task, and 11% in coding tasks. In addition to its novel architecture, evaluation results demonstrate that PLM outperforms existing small language models trained on publicly available data while maintaining the lowest number of activated parameters. Furthermore, deployment across various edge devices, including consumer-grade GPUs, mobile phones, and Raspberry Pis, validates PLM's suitability for peripheral applications. The PLM series models are publicly available at https://github.com/plm-team/PLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12167 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing Deng, Cheng Sun, Luoyang Jiang, Jiwen Zeng, Yongcheng Wu, Xinjian Zhao, Wenxin Xiao, Qingfa Wang, Jiachuan Li, Haoyang Chen, Lei Ni, Lionel M. Zhang, Haifeng Wang, Jun Computation and Language I.2.7 While scaling laws have been continuously validated in large language models (LLMs) with increasing model parameters, the inherent tension between the inference demands of LLMs and the limited resources of edge devices poses a critical challenge to the development of edge intelligence. Recently, numerous small language models have emerged, aiming to distill the capabilities of LLMs into smaller footprints. However, these models often retain the fundamental architectural principles of their larger counterparts, still imposing considerable strain on the storage and bandwidth capacities of edge devices. In this paper, we introduce the PLM, a Peripheral Language Model, developed through a co-design process that jointly optimizes model architecture and edge system constraints. The PLM utilizes a Multi-head Latent Attention mechanism and employs the squared ReLU activation function to encourage sparsity, thereby reducing peak memory footprint during inference. During training, we collect and reorganize open-source datasets, implement a multi-phase training strategy, and empirically investigate the Warmup-Stable-Decay-Constant (WSDC) learning rate scheduler. Additionally, we incorporate Reinforcement Learning from Human Feedback (RLHF) by adopting the ARIES preference learning approach. Following a two-phase SFT process, this method yields performance gains of 2% in general tasks, 9% in the GSM8K task, and 11% in coding tasks. In addition to its novel architecture, evaluation results demonstrate that PLM outperforms existing small language models trained on publicly available data while maintaining the lowest number of activated parameters. Furthermore, deployment across various edge devices, including consumer-grade GPUs, mobile phones, and Raspberry Pis, validates PLM's suitability for peripheral applications. The PLM series models are publicly available at https://github.com/plm-team/PLM. |
| title | PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2503.12167 |