Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.22101 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915576574640128 |
|---|---|
| author | Behdin, Kayhan Song, Qingquan Vasudevan, Sriram Sheng, Jian Ma, Xiaojing Zhou, Z Zhu, Chuanrui Li, Guoyao Nguyen, Chanh Ghosh, Sayan Sang, Hejian Baarzi, Ata Fatahi Ramachandran, Sundara Raman Wang, Xiaoqing Lan, Qing S, Vinay Y Guo, Qi Johnson, Caleb Wang, Zhipeng Borisyuk, Fedor |
| author_facet | Behdin, Kayhan Song, Qingquan Vasudevan, Sriram Sheng, Jian Ma, Xiaojing Zhou, Z Zhu, Chuanrui Li, Guoyao Nguyen, Chanh Ghosh, Sayan Sang, Hejian Baarzi, Ata Fatahi Ramachandran, Sundara Raman Wang, Xiaoqing Lan, Qing S, Vinay Y Guo, Qi Johnson, Caleb Wang, Zhipeng Borisyuk, Fedor |
| contents | Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search. However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements. In this work, we present lessons and efficiency insights from developing a purely text-based decoder-only Small Language Model (SLM) for a semantic search application at LinkedIn. Particularly, we discuss model compression techniques such as pruning that allow us to reduce the model size by up to $40\%$ while maintaining the accuracy. Additionally, we present context compression techniques that allow us to reduce the input context length by up to $10$x with minimal loss of accuracy. Finally, we present practical lessons from optimizing the serving infrastructure for deploying such a system on GPUs at scale, serving millions of requests per second. Taken together, this allows us to increase our system's throughput by $10$x in a real-world deployment, while meeting our quality bar. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_22101 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search Behdin, Kayhan Song, Qingquan Vasudevan, Sriram Sheng, Jian Ma, Xiaojing Zhou, Z Zhu, Chuanrui Li, Guoyao Nguyen, Chanh Ghosh, Sayan Sang, Hejian Baarzi, Ata Fatahi Ramachandran, Sundara Raman Wang, Xiaoqing Lan, Qing S, Vinay Y Guo, Qi Johnson, Caleb Wang, Zhipeng Borisyuk, Fedor Information Retrieval Machine Learning Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search. However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements. In this work, we present lessons and efficiency insights from developing a purely text-based decoder-only Small Language Model (SLM) for a semantic search application at LinkedIn. Particularly, we discuss model compression techniques such as pruning that allow us to reduce the model size by up to $40\%$ while maintaining the accuracy. Additionally, we present context compression techniques that allow us to reduce the input context length by up to $10$x with minimal loss of accuracy. Finally, we present practical lessons from optimizing the serving infrastructure for deploying such a system on GPUs at scale, serving millions of requests per second. Taken together, this allows us to increase our system's throughput by $10$x in a real-world deployment, while meeting our quality bar. |
| title | Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search |
| topic | Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2510.22101 |