MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917238015000576 |
|---|---|
| author | Li, Guoyao He, Ran Jing, Shusen Behdin, Kayhan Wang, Yubo Ramachandran, Sundara Raman Nguyen, Chanh Sheng, Jian Ma, Xiaojing Zhu, Chuanrui Vasudevan, Sriram Wu, Muchen Ghosh, Sayan Su, Lin Song, Qingquan Wang, Xiaoqing Wang, Zhipeng Lan, Qing Chen, Yanning Wu, Jingwei Simon, Luke Zhang, Wenjing Guo, Qi Borisyuk, Fedor |
| author_facet | Li, Guoyao He, Ran Jing, Shusen Behdin, Kayhan Wang, Yubo Ramachandran, Sundara Raman Nguyen, Chanh Sheng, Jian Ma, Xiaojing Zhu, Chuanrui Vasudevan, Sriram Wu, Muchen Ghosh, Sayan Su, Lin Song, Qingquan Wang, Xiaoqing Wang, Zhipeng Lan, Qing Chen, Yanning Wu, Jingwei Simon, Luke Zhang, Wenjing Guo, Qi Borisyuk, Fedor |
| contents | Large language models (LLMs) excel at capturing semantic nuances and therefore show impressive relevance ranking performance in modern recommendation and search systems. However, they suffer from high computational overhead under industrial latency and throughput requirements. In particular, cross-encoder ranking systems often create long context prefill-heavy workloads, as the model has to be presented with the user, query and item information. To this end, we propose MixLM, a novel LLM-based ranking framework, which significantly improves the system throughput via reducing the input context length, while preserving the semantic strength of cross-encoder rankers. In contrast to a standard ranking system where the context is presented to the model as pure text, we propose to use mix-interaction, a mixture of text and embedding tokens to represent the input. Specifically, MixLM encodes all items in the catalog into a few embedding tokens and stores in a nearline cache. The encoded item descriptions are used during online inference, effectively reducing the item length from a few thousand text tokens to a few embedding tokens. We share insights from deploying our MixLM framework to a real-world search application at LinkedIn, including a detailed discussion of our training pipelines, as well as a thorough analysis of our online serving infrastructure optimization. With the same latency budget and on-par relevance metrics, MixLM increased throughput by 10.0x comparing with strong baselines, 75.9x over full-text LLM rerankers. The efficiency gains delivered by MixLM enabled full-traffic deployment of LLM-powered search, which resulted in a significant 0.47\% increase in Daily Active Users (DAU) in online A/B tests. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_07846 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction Li, Guoyao He, Ran Jing, Shusen Behdin, Kayhan Wang, Yubo Ramachandran, Sundara Raman Nguyen, Chanh Sheng, Jian Ma, Xiaojing Zhu, Chuanrui Vasudevan, Sriram Wu, Muchen Ghosh, Sayan Su, Lin Song, Qingquan Wang, Xiaoqing Wang, Zhipeng Lan, Qing Chen, Yanning Wu, Jingwei Simon, Luke Zhang, Wenjing Guo, Qi Borisyuk, Fedor Information Retrieval Artificial Intelligence Computation and Language Machine Learning Large language models (LLMs) excel at capturing semantic nuances and therefore show impressive relevance ranking performance in modern recommendation and search systems. However, they suffer from high computational overhead under industrial latency and throughput requirements. In particular, cross-encoder ranking systems often create long context prefill-heavy workloads, as the model has to be presented with the user, query and item information. To this end, we propose MixLM, a novel LLM-based ranking framework, which significantly improves the system throughput via reducing the input context length, while preserving the semantic strength of cross-encoder rankers. In contrast to a standard ranking system where the context is presented to the model as pure text, we propose to use mix-interaction, a mixture of text and embedding tokens to represent the input. Specifically, MixLM encodes all items in the catalog into a few embedding tokens and stores in a nearline cache. The encoded item descriptions are used during online inference, effectively reducing the item length from a few thousand text tokens to a few embedding tokens. We share insights from deploying our MixLM framework to a real-world search application at LinkedIn, including a detailed discussion of our training pipelines, as well as a thorough analysis of our online serving infrastructure optimization. With the same latency budget and on-par relevance metrics, MixLM increased throughput by 10.0x comparing with strong baselines, 75.9x over full-text LLM rerankers. The efficiency gains delivered by MixLM enabled full-traffic deployment of LLM-powered search, which resulted in a significant 0.47\% increase in Daily Active Users (DAU) in online A/B tests. |
| title | MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction |
| topic | Information Retrieval Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2512.07846 |