Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.14305 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911232518258688 |
|---|---|
| author | Behdin, Kayhan Fatahibaarzi, Ata Song, Qingquan Dai, Yun Gupta, Aman Wang, Zhipeng Tang, Shao Sang, Hejian Dexter, Gregory Zhu, Sirou Zhu, Siyu Dharamsi, Tejas Kothapalli, Vignesh Fu, Zhoutong Cao, Yihan Hsu, Pin-Lun Borisyuk, Fedor Pillai, Natesh Simon, Luke Mazumder, Rahul |
| author_facet | Behdin, Kayhan Fatahibaarzi, Ata Song, Qingquan Dai, Yun Gupta, Aman Wang, Zhipeng Tang, Shao Sang, Hejian Dexter, Gregory Zhu, Sirou Zhu, Siyu Dharamsi, Tejas Kothapalli, Vignesh Fu, Zhoutong Cao, Yihan Hsu, Pin-Lun Borisyuk, Fedor Pillai, Natesh Simon, Luke Mazumder, Rahul |
| contents | Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications, from search and recommendation systems to generative tasks. Although scaling laws indicate that larger models generally yield better generalization and performance, their substantial computational requirements often render them impractical for many real-world scenarios at scale. In this paper, we present a comprehensive set of insights for training and deploying small language models (SLMs) that deliver high performance for a variety of industry use cases. We focus on two key techniques: (1) knowledge distillation and (2) model compression via structured pruning and quantization. These approaches enable SLMs to retain much of the quality of their larger counterparts while significantly reducing training/serving costs and latency. We detail the impact of these techniques on a variety of use cases in a large professional social network platform and share deployment lessons, including hardware optimization strategies that improve speed and throughput for both predictive and reasoning-based applications in Recommendation Systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14305 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems Behdin, Kayhan Fatahibaarzi, Ata Song, Qingquan Dai, Yun Gupta, Aman Wang, Zhipeng Tang, Shao Sang, Hejian Dexter, Gregory Zhu, Sirou Zhu, Siyu Dharamsi, Tejas Kothapalli, Vignesh Fu, Zhoutong Cao, Yihan Hsu, Pin-Lun Borisyuk, Fedor Pillai, Natesh Simon, Luke Mazumder, Rahul Information Retrieval Machine Learning Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications, from search and recommendation systems to generative tasks. Although scaling laws indicate that larger models generally yield better generalization and performance, their substantial computational requirements often render them impractical for many real-world scenarios at scale. In this paper, we present a comprehensive set of insights for training and deploying small language models (SLMs) that deliver high performance for a variety of industry use cases. We focus on two key techniques: (1) knowledge distillation and (2) model compression via structured pruning and quantization. These approaches enable SLMs to retain much of the quality of their larger counterparts while significantly reducing training/serving costs and latency. We detail the impact of these techniques on a variety of use cases in a large professional social network platform and share deployment lessons, including hardware optimization strategies that improve speed and throughput for both predictive and reasoning-based applications in Recommendation Systems. |
| title | Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems |
| topic | Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2502.14305 |